Skip to main content

Full text of "Make it So"

See other formats


Interaction Design Lessons from Science Fiction 


foreword by Bruce Sterling 



Nathan Shedroff and Christopher Noessel 

Rosenfeld Media 
Brooklyn, New York 


To my nieces, Aleksandra and Isabella, who have yet to see their first sci-fi. 
However, I have big plans for them and plenty of time to combat Barbie. 

—Nathan Shedroff 

To my nieces, nephews, and goddaughters: Hunter, Abby, Ava, Kaili, Andrea, 
Craig Jr., and Evan; and to my little, forthcoming boy (and any more to 
come). The vision of the future is increasingly in your hands. 

—Chris Noessel 


Being an interaction designer colors how you watch science fiction. Of 
course you're enjoying all of the hyperspacey, laser-flinging, computer- 
hacking action like everyone else, but you can't help but evaluate the 
interfaces when they appear. You are curious if they'll disable the tractor 
beam in time, but you also find yourself wondering, Could it really work that 
way? Should it work that way? How could it work better? And, of course, Can 
I get the interfaces I design in my own work to be this cool or even cooler? 

We asked ourselves these questions with each new TV show and each new 
film we watched, and we realized that for every eye-roll-worthy moment of 
technological stupidity, there are genuine lessons to be learned— practical 
lessons to be drawn from the very public, almost outsider-art interfaces that 
appear in the more than 100 years of sci-fi cinema and television. Then we 
wondered what we would learn from looking at not just one or even a dozen 
of them but as many as we could. 

This book is the result of that inquiry, an analysis of interfaces in sci-fi films 
and TV shows, with lessons that interface and interaction designers can use 
in their real-world practice. We've learned a great deal in writing it, and we 
want to share those lessons with you. 

Who Should Read This Book? 

We have written this book principally for interface designers interested 
in learning best practices from sci-fi, understanding sci-fi's role in design 
history, and using sci-fi interfaces in their own work. 

If you're a sci-fi fan with an interest in interface design, use this book to explore 
your favorite movies and TV shows more deeply and to discover new ones. 

If you make sci-fi, you can learn how the interfaces you create are evaluated 
by audiences and influence real-world developers. 

Similarly, individuals interested in media theory through the perspective of 
sci-fi can find insights here, though a more thorough and deep discussion of 
theory will have to wait for more research. 

What's in This Book? 

To make the material easily accessible, we've organized the discussions in 
two sections: the first examines the elements of user interfaces in sci-fi, and 
the second looks at how these interfaces are used to assist basic human 
activities such as communication and learning. 

Discussing interface elements first should make it clear where to find 
information, examples, and lessons pertaining to individual user interface 
components. These deal with inputs and outputs. Lots of examples can be 
found throughout sci-fi for each of these, but we've chosen some of the most 
interesting and unique. 

The second section focuses on things people do. This content is organized 
around the flow of activities and the system interactions that support users' 
goals. There's even a chapter on sex-related systems, of which there are more 
than you might at first think, and which reveal some surprisingly applicable 
lessons to everyday, less titillating work. 

All of the lessons and opportunities in the book have been gathered in an 
appendix for quick reference. 

What Comes with This Book 

There is a lot of material in this book, but we've still only scratched the 
surface. Lou Rosenfeld has been generous in giving us so much space, 
but there is a lot that couldn't be included, some of which is available on 
the book's companion website, There we'll be 
adding material as new films and TV series are released, a list of all of the 
titles we've reviewed so far, as well as links to where you can buy or rent 
titles, or watch clips. We're in the process of adding more detailed reviews 
of particular sci-fi interfaces, our extensive tag cloud, larger versions of the 
images used in the book, and more. 

How to Use This Book 


The topic of this book is a fun idea, but how 
is science fiction relevant to design? 

Design and science fiction do much the same thing. Sci-fi uses characters 
in stories to describe a possible future. Similarly, the design process uses 
personas in scenarios to describe a possible interface. They're both fiction. 
Interfaces only become fact when a product ships. The main differences 
between the two come from the fact that design mainly proposes what it 
thinks is best, and sci-fi is mostly meant to entertain. But because sci-fi can 
envision technology farther out, largely freed from real-world constraints, 
design can look to it for inspiration and ideas about what can be done today. 
See Chapters 1 and 14. 

Do you distinguish between science fiction and sci-fi? 

In a 1997 article, Harlan Ellison claimed the term "science fiction" for the 
genre of story that is concerned with science and "eternal questions," with 
an implied focus on literature. 1 We wanted to look at interfaces, and this 
led us quite often into that other category of story that he characterized as a 
"debasement" and "a simplistic, pulp-fiction view of the world" called "sci-fi." 
We don't entirely agree with his characterization, and it's true that we didn't 
look at literature for this project, so we don't make the same distinction. We 
just use sci-fi as an abbreviation for science fiction to save space. Hopefully 
Mr. Ellison won't be too mad. 

Where is [insert an example from sci-fi here]? 

To misquote Douglas Adams: Sci-fi is big. Really big. We couldn't get to 
everything, and we didn't have the room to include everything we got to. 
Fortunately, many sci-fi examples build on very similar ideas. Sometimes we 
passed over one example in favor of another that might be more well known 
or, alternatively, we included an unsung one that deserved some credit. 
Most of what we've reviewed is sci-fi from the United States, but we've also 
ventured into sci-fi from other countries. Even given what we've managed to 
achieve, we've barely scratched the surface. You can find additional material 
on our website: 

1 Ellison, Harlan. (1997, April 7). Strangers in a strange land. Newsweek. 

Why didn't you talk about [insert interaction 
design principle here]? 

The lessons are derived from sci-fi, not the other way around. If no example 
in the survey pointed us toward, say, Fitts's Law, then it doesn't appear, and 
some principles didn't make the final cut due to space constraints. Another 
style of investigation would have been to write a textbook on interaction or 
interface design using only examples from sci-fi, which would be interesting, 
but isn't this project. 

Wouldn't this have worked better as a movie 
or an ebook that can play video clips? 

Because our lessons and commentary involve moments from movies and 
television, it's a little problematic to publish them in a medium that doesn't 
allow us to show these interfaces in action. But because our focus was on 
studying interfaces and deriving lessons, we've started with media that would 
work best for later reference: traditional book, ebook, and website. If you're 
eager to see some of these interfaces in action, certainly check out the original 
movies or TV shows, or come to one of the workshops and lectures we give on 
the subject, where we share relevant clips. And be assured that we're exploring 
alternative media for these lessons and ideas next. 

These interfaces weren't designed to be studied or for 
users in the real world. Aren't you being a little unfair? 

Indeed, we are using real-world criteria for interfaces that aren't in the 
real world — the vast majority of which aren't meant to be. But as fans and 
designers, we can't help but bring a critical eye to bear on the sci-fi we watch, 
and with most of the world becoming more technologically savvy as time 
goes on, audiences will become so, too. But it's the "outsider" nature of these 
interfaces that make them fascinating to study, as their creators produce 
both blunders and inspired visions. 

What was the most interesting thing you 
discovered when writing the book? 

We were surprised at how productive it was to investigate the "bad" 
interfaces. The "good" interfaces often serve as reminders of principles with 
which we are already familiar. Sometimes they are inspiring. But the "bad" 
interfaces, because they still worked at a narrative level, revealed the most 
surprising insights through the process of "apology," discussed in Chapter 1. 

Frequently Asked Questions 

What was left on the editing room floor? 

One of our early ideas for the book was to include interviews with sci-fi 
makers and science practitioners. The interviews didn't make it into the final 
iteration of the book, but these people gave their time and shared much with 
us, and we'd like to acknowledge them individually with special thanks: 
Douglas Caldwell, Mark Coleran, Mike Fink, Neil Huxley, Dean Kamen, Joe 
Kosmo, David Lewindowsky, Jerry Miller, Michael Ryman, Rpin Suwannath, 
and Lee Weinstein. 

Additionally, we had early draft chapters on sci-fi doors, chemical interfaces, 
weapons, and spacesuits/spaceships. Early reviews of the sheer size of the 
book forced us to make some hard choices. Perhaps in some future work we 
will be able to develop this content further, but for now it will have to wait. 

Why didn't you mention [insert title] more? 

Several movies and TV shows are incredibly seminal and culturally 
influential. Star Trek, Minority Report, and 2001: A Space Odyssey are three 
we can name off of the top of our heads. But we didn't want to lean too much 
on a small set of movies and shows. Rather, we wanted to use these examples 
for their most salient aspects, then branch out into other examples from the 
survey when the topic warranted. 

What about other speculative technology found in 
video games, futuristic commercials, or industry films? 

The hard-core genre nerds know that conversations about defining science 
fiction often lead to conversations about speculative fiction instead, which 
is a much broader topic of interest to us, but isn't the focus of this project. 
Anyone interested in these related media should read Chapter 14. 

Frequently Asked Questions 


How to Use This Book iv 

Frequently Asked Questions vi 

Foreword xvii 


Learning Lessons from Science Fiction 1 

What Is an Interface? 3 

Which Science Fiction? 3 

What Counts? 5 

Why Look to Fiction? 6 

The Database 7 

Finding Design Lessons 7 

The Shape of a Lesson 10 

Finding Inspiration in Science Fiction 11 

Let's Begin 13 



Mechanical Controls 15 

At First, Mechanical Controls Were Nowhere 16 

Then They Were Everywhere 17 
For a While, Mechanical Controls Started 

Disappearing 21 

Now They Coexist with Other Interfaces 24 

Mechanical Controls Are Used to Evoke Moods 26 

Mechanical Controls: Will We Come Full Circle? 27 


Visual Interfaces 29 

What Counts? 32 

Text-Based Interfaces 32 

Command-Line Interfaces 32 

Graphical User Interfaces 36 

Typography 36 

Glow 40 

Color 41 

Display Shape 50 

Layers and Transparency 51 

2V2D 54 

Grouped Controls 55 

File Management Systems 58 

Motion Graphics 62 

Visual Style 64 

The Hitchhiker's Guide to the Galaxy 65 

Final Fantasy 66 

The Chronicles of Rid dick 66 

The Incredibles 67 

Case Study: Star Trek's LCARS 68 

Visual Interfaces Paint Our Most Detailed 

Pictures of the Future 73 


Volumetric Projection 75 

What Counts? 76 

What Do Volumetric Projections Look Like? 78 

How Are Volumetric Projections Used? 81 

Communications 81 

Reinforcing Social Hierarchy 85 

Navigation 86 

Medical Imaging 87 

Real-World Problems 87 

Confusion 87 

Eyestrain 88 

Cropping 88 






Volumetric Projection Has Been Defined by Sci-Fi 





What Counts? 


The Canonical Gestural Interface: Minority Report 


Gesture Is a Concept That Is Still Maturing 


Hollywood's Pidgin 


1. Wave to Activate 


2. Push to Move 


3. Turn to Rotate 


4. Swipe to Dismiss 


5. Point or Touch to Select 


6. Extend the Hand to Shoot 


7. Pinch and Spread to Scale 


Direct Manipulation 


Gestural Interfaces Have a Narrative Point of View 


Gestural Interfaces: An Emerging Language 



Sonic Interfaces 


What Counts? 


Sound Effects 


Ambient Sound 


Directional Sound 


Music Interfaces 


Voice Interfaces 


Simple Voice Output 


Voice-Identification Interfaces 


Limited-Command Voice Interfaces 



Conversational Interfaces 121 

Sonic Interfaces: Hearing Is Believing 124 


Brain Interfaces 125 

Physically Accessing the Brain 126 

Invasive Brain Interfaces 126 

Noninvasive Brain Interfaces 127 

Disabling the Mind 131 

Two Directions of Information 132 

Writing to the Brain 132 

Reading from the Brain 138 

Telexperience 142 

Active Subjects 144 

Virtual Telepresence 144 

Actual Telepresence 148 

Manifesting Thought 149 

Having Virtual Sex 149 

Piloting a Spaceship 150 

Playing a Game 151 

Dismantling Two Sci-Fi Brain-Tech Myths 151 
Myth: Bra in- Affecting Interfaces Will 

Be Painful 151 
Myth: Knowledge Can Be Installed and 

Uninstalled Like Software 153 

Where Are the Thought Interfaces? 153 

Brain Interfaces: A Minefield of Myths 155 


Augmented Reality 157 

What Counts? 158 

Appearance 160 

Sensor Display 160 


Location Awareness 163 

Context Awareness 165 

Object Awareness 165 

Awareness of People 167 

Goal Awareness 171 

Goal: Flying Well 171 

Goal: Precise Targeting 172 

What's Missing? 176 

Augmented Reality Will Make Us Laser-Focused, 

Walking Encyclopedias 176 


Anthropomorphism 177 

Humanness Is Transferable to Nonhuman 

Systems 179 

Appearance 185 

Voice 186 

Audible Expressiveness 188 

Behavior 189 
Degrees of Agency: Autonomy and Assistance 190 
Anthropomorphism: A Powerful Effect That 

Should Be Invoked Carefully 195 



Communication 197 

Asynchronous versus Synchronous Communication 199 

Composing 199 

Playback 201 

Activating the System 202 

Specifying a Recipient 203 

Fixed Connection 203 




A Unique Identifier 


Stored Contacts 


Receiving a Call 




What We Don't See 




Monitoring the Connection 


Ending a Call 






What We Don't See 




What We Don't See 


Two More Functions 


Language Translation 




Communication: How We'll Be Talking Next 





Direct Download 


Psychomotor Practice 


Presentation Tools 


Reference Tools 


Machines to Think With 


Testing Interfaces 


Case Study: The Holodeck 


Psychomotor Training 






Machines to Think With 



Lessons Unique to the Holodeck 


What We Don't See 


Learning: Aiming for the Holodeck 





Assistive Medical Interfaces 


An Ounce of Prevention 








Autonomous Medical Interfaces 


Case Study: The Doctor 


Life and Death 


Assisting Birth 




Signaling Death 


Sci-Fi Medical Interfaces Are Focused Mainly 

on the Critical Situation 







Sex with Technology 






Virtual Partners 




Augmented Coupling 




Mediated Coupling 


The Interface Is Not the Sex 




What's Next? 309 

Using Sci-Fi 310 

More Than Sci-Fi 311 

And Sci-Fi to Come 313 

Appendix: Collected Lessons 

and Opportunities 315 

Credits 323 

Index 327 

Acknowledgments 346 

About the Authors 347 



They Made It So 

This book has accomplished a feat that's valuable and rare: it comprehends 
design and science fiction. Better yet, it's found specific areas where they are 
of practical use to one another. 

This is a design book, and meant for designers. It concerns itself with 
science fiction cinema. To my delight, it does this in a deft, thoughtful, and 
sympathetic way. 

Make It So never asks science fiction to be "scientific." More tactfully, 
it doesn't even ask that science fiction be "fictional." Instead, this book 
comprehends the benefits that science fiction can offer to designers. There 
aren't a lot, but there are some. Those benefits are all about making the 
unthinkable thinkable. "Cognitive estrangement," as we science fiction 
people call that in our trade. 

Make It So teaches designers to use science fiction as a designer's mood 
board. It's science fiction as an estranging design tool, a conceptual 
approach, best suited for blue-sky brainstorming, for calling the everyday 
into question, and for making the exotic seem practical. 

This approach allows designers to derive all kinds of exciting design benefits 
that science fiction never intended to bestow on designers. 

How do the authors do it? With a classic, people-centered design approach. 
They look and they listen. They are at ease with the creators of science fiction 
cinema, because they can enter into their worldview. 

Consider Georges Melies, that silent-film maestro of cinema's earliest days, 
that French stage magician turned movie fantasist. For most of us, Melies is 
a remote historical figure whose accented French name is hard to properly 
spell. He's of real, immediate use to Shedroff and Noessel. 

Even us science fiction writers — (I write novels, by the way) — we rarely derive 
any coherent inspiration from our remote spiritual ancestor, Georges Melies. 

But Shedroff and Noessel are able to enter into the Melies conceptual 
universe with all the attentive consideration that designers commonly grant 
to users. So the authors of this book can see that the best-known film of 
Georges Melies, A Trip to the Moon (Le voyage dans la lune), has no interfaces. 

That's the truth, of course — obviously a silent-film spaceship from 1902 has 
no interfaces, because the very concept of an "interface" didn't show up until 
the 1960s. However, it requires a design perspective to see past the frenetic 

razzle-dazzle on the silver screen and point that out. Melies was a major 
media pioneer, and yet he was interfaceless. 

Furthermore, this is an exciting and refreshing thing for a science fiction 
writer to read. Although Shedroff and Noessel don't intend to write their 
book for us science fiction creatives, I'd boldly say that they're every bit as 
useful to us as we could ever be to them. 

Melies had no interfaces. This startling realization blows the dust of the 
ages off of Melies and conveys a new sheen to his time-dulled glamour and 
wonderment. As soon as I read this, I put the text of Make It So aside— 
(because, to tell the truth, I was reading the book on a screen)— and I sought 
out and watched the Melies 1902 film on YouTube (on the same screen). 

The authors are correct. Try it for yourself! The characters in this Melies 
movie are inhabiting an attitude toward technology that's alien to us. Watch 
them go through their entirely mechanical design paradigm, all anvils and 
chalkboards. They have no push buttons, no rheostats, no dials, no screens, 
no return keys. They have no systematic abstraction of the forces that 
surround them, other than books and papers. They're on a sci-fi trip to the 
Moon to meet space aliens, and they might as well be paddling a steel canoe. 

How mind-stretching that realization is. 

Furthermore, Shedroff and Noessel gently suggest— (this book was written 
by designers, so they're very urbane, low key, and eager to be of service)— 
they suggest that, for an interface designer, the best way to look at a Melies 
spaceship is as a potential way forward. Not a historical curiosity, a thing 
frozen on aging film like a fossil in amber, but a potential future for interface 
design. What a fascinating thing to say! What if the controls of future 
spacecraft were so natural, so intuitive, so invisible, that they were Melies- 
like in their magical simplicity? 

Why has no science fiction writer yet written this scene? Where is the 
science fiction set within a gesture-controlled, augmented, and ubiquitous 
environment? I've often wondered that — but I know that it's difficult to 
conceive, it's hard to sketch out as any workable scenario. It never occurred 
to me such a high-tech situation might have the look and feel of Melies' 
fantasy movie: ritualized, formal, very gestural, everything tightly framed. 
It's a brilliant notion, though. It jolts that prospect from the remote to the 
immediate. Why, it's almost tangible. 

People commonly expect science fiction to be predictive. Shedroff and 
Noessel, to their credit, avoid that mistake. I happen to believe that science 


fiction often is predictive: but so what? If you successfully predicted 1975 
while you were writing in 1960, there's no reason why anyone nowadays 
would know or care about that. The works of science fiction that last are 
never accurate forecasts. They're compelling evocations— they're visionary 
grotesques, funhouse mirrors. 

That funhouse mirror is never accurate, yet it doesn't merely deceive either— it 
always bears its human intent to inspire wonderment, its innate need to capture 
the imagination. Sci-fi, even at its most analytic and mechanical, is always 
haunted, allusive, and esoteric. Sci-fi is like a Rorschach blot the size of a house. 

Make It So is like sci-fi film critique, but of a new kind: with kindly 
instructors equipped with a remote control and a freeze-frame. They 
deliberately break sci-fi cinema into its atomic design elements. 

It's wonderful how they waste no time with any stereotypical sci-fi 
criticism— the characters, the plot, the so-called political implications. 
Legions of other critics are eager to get after that stuff, whereas Shedroff and 
Noessel have created a lucid, well-organized design textbook. I recommend 
this textbook for class work. I can't doubt for a moment that contemporary 
students would be illuminated and grateful. 

Science fiction and design have a relationship: it's generally cordial, yet remote. 
Design cannot realize the fantasies of science fiction. Science fiction can't help 
design with all its many realistic problems. Design and science fiction were born 
in the same era, but they're not family: they're something like classmates. The 
two of them have different temperaments. Sometimes design is visionary and 
showy, and in sync with its classmate, sci-fi. At other times, design is properly 
concerned with its own issues of safety, utility, maintenance, and cost, areas 
where science fiction always stares moodily out the window. 

But eras appear when the technological landscape changes quickly and 
radically, and design and science fiction are dragged along in tandem. 

Interface design is one of those areas, and inhabiting one of those times. 
Science fiction is unlikely to be of great help in the task of giving form to a 
vase. However, interface design requires a certain mental habit of speculative 
abstraction. That isn't science fiction, but it's not so far as all that. "Interaction 
design" is quite similar to "interface design" — interaction designers are 
obsessed with boxes and arrows, not clay or foamcore. When design genuinely 
needs to be conceptual and abstract, science fiction can put a face on that. 
Science fiction can embody and literalize that, it can tell that story. 

Somewhere over the horizon, beckoning at us, is "experience design." 


This is something we associate with computer games and thrill rides and 
imaginary Star Trek holodecks, but it will likely have something to do 
with tomorrow's cloudy, post-cybernetic environments. When it comes to 
battling those obscure future phantoms, design and sci-fi are in a masked- 
wrestler tag-team match. It's us— an unlikely duo— against that, a futuristic 
prospect. We're gonna pin that phantom to its augmented, ubiquitous mat 
someday, but it's gonna take some sweat and bruises first. 

It will take sweat, bruises, and also some intense blue-sky thinking. Some 
of that is already visible within modern big-budget sci-fi movies— Minority 
Report, Iron Man, they're full of pricey interface thrills, just as these authors 
will show you. But, increasingly and interestingly, a great deal of that 
necessary conceptual work will never appear in big movies, but in small-scale, 
atelier-like, design-centered videos. It will appear on this screen, not the big 
silver screen but this interactive, designed screen, the screen where I read this 
book, and where I saw that public-domain Melies movie. This is no accident. 

I like to call this small-scale, speculative work "design fiction." Design 
fiction is the deliberate use of diegetic prototypes to suspend disbelief 
about change. There's a lot of "diegetic prototyping" going on now, and that 
situation has come to exist, primarily, because of interface design. It is a 
consequence of interfaces built for the consumption and creation of what 
used to be called "text" and "film." 

The movies, and television, as analog industries, as 20th-century 
commercial entities, would never have done that on their own. They would 
never have imagined the viral creation and global spread of speculative 
videos about futuristic products and services. This did not fit their business 
model. It was outside their paradigm. 

Even science fiction writers didn't imagine that. But it's an area of great ferment: 
these attempts to employ digital media to convince people to transform 
conceptual things into real things. I see it every day. Interface design is powerful. 
It changed my life, and I expect it to transform my future life even more so. 
People who read this book will be better equipped to undertake that effort. 

I never imagined that I would be reading a book like this, or that it would be 
this good. 

— Bruce Sterling 
Turin, Italy, May 2012 



Lessons from 
Science Fiction 

What Is an Interface? 


Which Science Fiction? 


What Counts? 


Why Look to Fiction? 


The Database 


Finding Inspiration in Science Fiction 


Let's Begin 


Science fiction and interface design were made for each other. An 
interface is the primary way a sci-fi audience understands how the 
characters in stories use nifty, speculative technologies. And interface 
design (cautiously) loves to see fresh ideas about potential technologies 
unbound by real-world constraints writ large in the context of exciting stories. 

This gives interfaces in the real world an interesting and evolving 
relationship with interfaces seen in sci-fi. With technology advancing 
quickly in the real world, sci-fi makers must continually invent more 
fantastic technologies with newer and more exciting interfaces. As 
audiences around the world become more technologically sophisticated, 
sci-fi makers must go to greater lengths to ensure that their interfaces are 
believable and engaging. And as users compare sci-fi to the interfaces they 
use every day, they're left to dream about the day when their technology, too, 
will become indistinguishable from magic. 

But the relationship between the two is also an unfair one. Sci-fi can use 
smoke, mirrors, and computer-generated imagery to make things look 
incredibly exciting while ignoring practical constraints like plausibility, 
usability, cost, and supporting infrastructure. A sci-fi interface is rarely 
shown for more than a few seconds, but we use real-world interfaces, such as 
word-processing or spreadsheet software, for hours on end, year after year. 
Interfaces in the real world must serve users in an unforgiving marketplace, 
where lousy interfaces can quickly kill a product. But those same users 
might overlook a lousy interface in a great movie with no questions asked. 

The relationship is also one of reciprocal influence. Every popular real-world 
interface adds to what audiences think of as "current" and challenges sci-fi 
interface makers to go even further. Additionally, as audiences become 
more technologically literate, they come to expect interfaces that are 
more believable. Sci-fi creators are required to pay more attention to the 
believability of these interfaces, otherwise audiences begin to doubt the 
"reality" created, and the story itself becomes less believable. This raises 
the stakes for sci-fi. Real-world interface designers are wise to understand 
this dynamic, because audience expectations can work the same way for 
their creations. 

Make It So: Interface Lessons from Science Fiction investigates this 
relationship to find a practical answer to this question: What can real-world 
interface designers learn from the interfaces found in science fiction? 

To begin to answer this question, we first need to define what we mean by 
"interface" and "science fiction." 

Chapter 1 

What Is an Interface? 

The term interface can refer to a number of different things, even in the 
world of software. In this book, we use it specifically to mean user interface 
as it pertains to human-computer interaction. With most people's computer 
experience centering on mobile phones, laptop computers, and desktop 
computers, familiar examples would be the keyboards, mice, touch screens, 
audible feedback, and screen designs of these objects. We generally mean 
the same thing in sci-fi, though the inputs and outputs of speculative 
technology stray pretty quickly from these familiar references. For example, 
does a hologram or volumetric projection count as a screen? And where's the 
keyboard in a Star Trek tricorder? 

A more abstract definition allows us to look at these fictional technologies 
and speak to the right parts. The working definition we're using to define an 
interface is "all parts of a thing that enable its use." This lets us confidently 
address the handle and single button of a lightsaber as the interface, 
while not having to address the glowing blade in the same breath. While 
researching this book, we've had this definition in mind. 

This definition leads us to include some aspects of interfaces that we 
might not ordinarily consider in a more conventional, screen-and-mouse 
definition. For instance, the handle of a blaster is three-dimensional and 
doesn't do anything on its own, but if that's how you hold it, it's definitely 
part of the interface. This means that, over the course of our investigation, 
we may touch on issues of industrial design. 

Similarly, we may run into problems with the organization of information 
that we see on sci-fi screens, which is part of what enables use. Does the 
character's screen make sense? Addressing this question means we may 
touch on issues of information design. 

We may also need to look at the connection between the actions a 
character performs and the output they see— their intent and the outcome. 
Interactions over time are a critical element of the interface, and this 
requires us to evaluate the interaction design. 

The "interface," then, is the combination of all of these aspects, though we 
try to focus on the most novel, fundamental, or important of them. 

Which Science Fiction? 

Science fiction is a huge genre. It would take years and years to read, watch, 
and hear it all. Even before we had a chance to step back and study all we've 
taken in, there would be even more new material requiring our attention. (Oh, 
but for a Matrix-style uploader: "I know all of sci-fi!") Fortunately, looking 
specifically for interfaces in sci-fi reduces the number of candidates for this 
survey. The first way it does so is through the media of sci-fi. 

Learning Lessons from Science Fiction 

For the purposes of this investigation, to evaluate an interface we have to see 
and hear it. This is so that we can understand what the user must take into 
account when trying to make sense of it. Literature and books often describe 
the most important parts of interfaces, but often fail to describe details, 
which each reader might imagine quite differently; this makes interfaces 
described in writing nearly impossible to evaluate. Take this description 
from H. G. Wells's The Time Machine (1895), for example: 

"This little affair," said the Time Traveller, resting his elbows upon 
the table and pressing his hands together above the apparatus, "is 
only a model. It is my plan for a machine to travel through time. 
You will notice that it looks singularly askew, and that there is an 
odd twinkling appearance about this bar, as though it was in some 
way unreal." He pointed to the part with his finger. "Also, here is 
one little white lever, and here is another." 

The Medical Man got up out of his chair and peered into the thing. 
"It's beautifully made," he said. 

"It took two years to make," retorted the Time Traveller. Then, 
when we had all imitated the action of the Medical Man, he said: 
"Now I want you clearly to understand that this lever, being 
pressed over, sends the machine gliding into the future, and this 
other reverses the motion. This saddle represents the seat of a 
time traveller. Presently I am going to press the lever, and off the 
machine will go. It will vanish, pass into future Time, and disap- 
pear. Have a good look at the thing. Look at the table, too, and 
satisfy yourselves that there is no trickery. I don't want to waste 
this model, and then be told I'm a quack." 

Although this description of the interface is useful, it's incomplete. Are the 
levers in easy reach? Are they a meter long or a couple of millimeters? Do 
they press away from the Traveller, toward him, or parallel to his chest? How 
are they labeled? Are there forces in effect while time traveling that make the 
machine easier or harder to operate? The archetype that the reader imagines 
is probably sufficient for the purposes of the story, but to really evaluate and 
learn from it requires much more detail. For these reasons, we decided not to 
consider interfaces from written science fiction. 

For similar reasons, we need to see the character's use of an interface over 
time. If we were to evaluate only still pictures, we might not know, for example, 
how information appears on a screen, or what sounds provide feedback, or 
whether a button is pressed momentarily or held in position. Comic books, 
concept art, and graphic novels sometimes supply this information, but 
unless their creators provide unusual levels of detail, the resolution is too 
crude and the interstices of time make it difficult to get a complete sense of 
how an interface is intended to work. Due to this complication, we have not 
considered comic books, graphic novels, or concept art either. 

Chapter 1 

And finally, even with visual depictions across time, as in animation, the 
interface needs to remain consistent from scene to scene. Otherwise, we 
would have to interpret the intended interface (much like with written 
sci-fi) and risk conflicting, confusing conclusions. For this reason, we've 
mostly avoided hand-drawn, animated interfaces like those found in anime 
or Futurama. Of course, these problems can crop up in film and TV sci-fi, 
too, but for most 3D-animated or live-action interfaces, the depictions are 
consistent enough to evaluate them as single systems. 

These three requirements— that the medium be audiovisual, time-based, 
and consistent— leave us with 3D-animated or live-action sci-fi for cinema 
and television. Sci-fi in these media give us candidate interfaces that we can 
examine for design lessons. 

What Counts? 

A trickier question is, What counts as science fiction? Of course, some are 
obvious, such as tales of people racing spaceships between planets, shooting 
ray guns at villains, and making out with the comelier of the aliens. But 
what about the spy genre? From self-destructor bags to pen guns to remote- 
controlled Aston Martins, they certainly feature speculative technology. And 
what about steampunk fiction and superhero movies? Or slapstick comedy 
sci-fi like Spaceballs? Media properties in each of these genres contain all sorts 
of gadgets with interfaces that could bear some kind of examination. 

These are good questions, but ultimately we've avoided the academic pursuit 
of defining science fiction and tried to remain purposefully agnostic. 
Generally speaking, if the Internet Movie Database ( defines 
a movie or a TV show as science fiction, it has been up for consideration. 

Occasionally we've looked outside of sci-fi, comparing interfaces for real-world 
systems, products, and prototypes with similar goals or functions, when 
it's relevant and space allows. We've also looked to speculative interfaces 
in notable industrial films from companies that posit new computing 
experiences. In this sense, it might be most accurate to say we're looking at 
speculative fiction, but because most people haven't heard this term, we use the 
common term sci-fi throughout the book. 

We have watched and analyzed a great many sci-fi properties, but there is 
no way we could cover everything. We've tried to include the most notable 
and influential properties, both in the sense of influencing design as well 
as culture as a whole. However, it's likely we haven't yet covered something 
interesting or dear to all readers. The accompanying website to this book,, includes notes and more extensive analysis of 
many more properties than there is room for here. In addition, the website 
will serve as a place for ongoing updates and commentary on new properties 
after the publication of this book. 

Learning Lessons from Science Fiction 

Why Look to Fiction? 

With a working category of sci-fi and having decided what to focus on, we 
next ask the question: Why look to fiction for design lessons at all? How can 
it inform our non-fictional, real-world design efforts? 

One answer is that, whether we like it or not, the fictional technology seen in 
sci-fi sets audience expectations for what exciting things are coming next. 
A primary example is the Star Trek communicator, which set expectations 
about mobile telephony in the late 1960s, when the audience's paradigm was 
still a combination of walkie-talkie and the Princess phone tethered to a 
wall by its cord. Though its use is a little more walkie-talkie than telephone, 
it set the tone for futuristic mobile communications for viewers of prime- 
time television. Exactly 30 years later, Motorola released the first phone 
that consumers could flip open in the same way the Enterprises officers did 
(Figure 1.1). The connection was made even more apparent by the product's 
name: the StarTAC. The phone was a commercial success, arguably aided by 
the fact that audiences had been seeing it promoted in the form of Star Trek 
episodes and had been pretrained in its use for three decades. In effect, the 
market had been presold by sci-fi. 

Another answer is that with media channels proliferating and specializing, 
common cultural references are becoming harder and harder to come by. 
Having common touchstones helps us remember design lessons and discuss 
ideas with each other. Sci-fi is a very popular genre, and the one in which 
speculative technology is seen most often. If you want to discuss an existing 
technology, you can reference a real-world interface. But to discuss future 
technologies, it's easier to reference a movie than to try to define it a priori: 
"Kinect is, you know, kind of like that interface from Minority Report, but 
for gaming." 

FIGURE 1.1a,b 

Star Trek: The Original Series (1966); the Motorola StarTAC (1996). 

Chapter 1 

A last answer is that interface makers in the real world and in sci-fi are, 
essentially, doing the same thing— creating new interfaces. In this sense, 
all design is fiction— at least until it gets built or is made available to users 
and customers. When designers create anything that isn't the real, final 
product that ships, they're creating speculative interfaces— fictions. Each 
wireframe, scenario, pencil sketch, and screen mockup says, "Here's how it 
might be," or even "Here's how it ought to be." Designers for each domain 
ask similar questions: Is this understandable? What's the right control for 
this action? What would be awesome? Although they ultimately work with 
different audiences, budgets, media options, goals, and constraints, the work 
is fundamentally similar. Each can learn something from the other. 

The Database 

Once we had a set of movies and TV properties to review (see the complete list 
online at, we watched and evaluated everything 
we could get our hands on. We entered screenshots and descriptions into a 
custom database, which formed the basis for our investigation. This database 
is also available on the website, where you can make your own contributions 
and see much of the content that could not fit into this book. 

Finding Design Lessons 

Armed with this tool, we then identified what we could learn from the 
interfaces. There are four ways we go about this. 

Bottom Up 

To learn lessons from the bottom up, we investigated an individual interface 
in detail. To do this, we need an interface whose use we understand and that 
has sufficient screen time to allow us to analyze its inputs, compare these 
with its outputs, and evaluate what works for the user in accomplishing his 
or her goal. If it doesn't work, we may still be able to learn a lesson from a 
negative example. If it does work, we can compare it to any similar interfaces 
we find in the real world to see what might translate. The things that can 
translate are captured as lessons, and we can later look for other examples 
in the survey that support or refute it. 

Top Down 

To examine the survey from the top down, we tagged each description in the 
database with meaningful attributes. The example in Figure 1.2 shows a set 
of tags for the write-up on the wall-mounted videophone seen in Metropolis. 

Learning Lessons from Science Fiction 

Description: Joh verifies that he's seeing the correct channel visually when he sees Grot's 
nervous pacing in camera view. Confident that he's calling the right place, Joh picks up a 
telephone handset from the device and reaches across to press a control on the right. In 
response, the lightbulbs on Grot's videophone begin to blink and, presumably (it's a silent 
film), make a sound. 

Tags: analog, calling, communication, dial, dials, filmmetaphor, hangingup, hangup, 
messages, printedoutput, telephone, telephony, tickertape, tuning, turningoff, videophone, 
wallmounted, wristroll, wristtwist 

FIGURE 1.2a-c 
Metropolis (1927). 

With the interfaces in the database tagged, we looked at the aggregated 
tag cloud to see what stood out. We then drilled down into the tags 
that appeared most often: glow, screen, red, blue, video, and holography 
(Figure 1.3). We then tried to explain why the tag appears so frequently, 
compared the interfaces similarly tagged, considered their commonalities 
and differences, and compared them with interfaces in the real world. 

. nonrcciBn^LunoHfi 

button bkMdentrfication 



r w K e SKlzJ I Is! WW au*>SWSS? 

handheld JsL,^^ I ^^i— W*W telecommunication ««-. 

12g£% *KKE? U I nx V| Q"0 .cockpit 

translucent SGCU f IV M I II Q ** w anthropomorphism rrt* 
Tr^ncrS^nJ **"* UlUvJ keyboard Poirce W eapon - 
iransparGnt'"^ artitoiinteftgenrai 0^^* voji^netncdisplay w j r eiess 

arthicialintellgence I overlay ""VOIUmeUltaibpidy wJreleSS ™ 


"■TODOl warning 

taTjSEig noiocjrsrn a ™ 

r^ Videophone rjrnOlOQrapny mn p| ^'"interface tele 

nightcontrois jja telephony m ^hiAtan ^" V? 9estural HST£» 
v^- access t*. . .PUSflPUTOn headsupdisplay t«i*w*> aor ** 

""" identification 


This tag cloud, created using tools at, illustrates the major 

top-down themes. 

Chapter 1 

Chasing Similarities 

Another way to glean design lessons from the survey is to notice and pursue 
personally observed similarities between properties. For example, fans of 
gestural interfaces may have noticed similarities between the controlling 
gestures appearing in completely different movies and TV shows, from 
different writers and even different studios. What's going on here? Since 
there's not a gesture czar calling the shots, what's underneath these 
similarities? Are they coming from existing interfaces, common sense, or 
somewhere else? Investigating questions like these is something of a top- 
down approach, but it comes from pursuing particular questions rather 
than letting the questions emerge from the tags. (See Chapter 5 for some of 
our answers to these questions about gestural interfaces.) 


One of the most rewarding techniques is apologetics (we're borrowing the 
term from theology). When we found an interface that couldn't work the way 
it was shown, we looked for ways to "apologize" for it; that is, we thought of 
ways that the interface could work the way it was depicted. In a few cases, 
this led to some interesting insights about the way technology should work. 

One example of this comes from 2001: A Space Odyssey. From an Earth- 
orbiting space station, Dr. Floyd has a videophone conversation with his 
daughter back on Earth. During the scene, we see the young girl's hands 
mash on the keypad of the phone, but the call isn't interrupted (Figure 1.4). 
Although this may have been an oversight on the director's part, it is 
nonetheless the way the system should work. If the system knows that a 
child is using it and the button mashing is likely unintentional, it should 
disregard these inputs and not interrupt the call. Although this presumes 
sophisticated technology and an interface idea even the film's producers 
probably didn't think about, we can still use this principle even as we work 
with our real-world technology today. 


2001: A Space Odyssey (1968). 

Learning Lessons from Science Fiction 

This technique, more than any of the others, may have pragmatic readers 
scratching their head, and asking if sci-fi interface designers really put as 
much thought into their creations as we have in examining them. 

It's entirely possible that they don't, that sci-fi interfaces are a product of pure 
inspiration, produced under tight deadlines with little time for research or 
careful reflection. But to be of use to us who are able to reflect on the interfaces 
we create, we have to examine them as if they were produced exactly as the 
designers intended them to be. It's a choice you have to make when writing 
critique, an issue referred to in literary circles as authorial intent. We chose to 
look at the interfaces without trying to reverse-engineer intent. If we didn't, 
we might get spun out on vicious cycles of second-guessing. 

We used all of these techniques in the development of this material. The 
bottom-up approach provided many individual lessons. The top-down 
approach provided a reliable path through the vast amount of material we 
had to work with, and provided much of the structure of the book. Chasing 
similarities resulted in a few particular chapters, like Volumetric Projection 
(Chapter 4) and Gesture (Chapter 5). Apologetics resulted in the most 
satisfying results from the material, though, because we had to use what 
worked right from a narrative stance— a human stance— to arrive at new 
interaction design ideas. We couldn't count on finding these opportunities 
in sci-fi, since we had to wait to find "mistakes," but we could take advantage 
of them when we did. 

The Shape of a Lesson 

When capturing lessons, our goal was to provide them in a useful format. 
We want them to be easily spotted as you read or skim through the material, 
so they are set off in green type. The titles of the lessons are written as 
unambiguous imperatives, so their intended lesson is clear. We've included 
a description in accessible language that calls out nuances, extends the 
examples, and describes when the lesson is applicable. 

Sometimes, the analysis points to something that wasn't seen in the survey. 
These particular lessons are called out as Opportunities, but are otherwise 
similar in appearance. 

Finally, we gathered together all of the lessons in an appendix at the back 
of the book so you can find a particular one more easily and consider them 
as a set. 

10 Chapter 1 

Finding Inspiration in Science Fiction 

In the year 2000, Douglas Caldwell was successfully petitioned by his 
teenage son to see the filml-iKen. Douglas wasn't really a fan of sci-fi, but 
wanted to spend time with his son, so he agreed to go. Watching the film, 
he was amazed to see a solution to a 2,000-year-old problem that he dealt 
with every day. 

In a scene near the climax, the X-Men are gathered around a large display 
surface, which looks something like a circular, metallic tabletop. As Cyclops 
describes the mission they are about to undertake, the map changes shape, 
as if it was made of hundreds of tiny pins, each rising and falling to form the 
topography needed (Figure 1.5). 

The reason this speculative technology was so important to Douglas was 
that he worked for the US Army Topographic Engineering Center. Part of 
his job was to create 3D maps and ship them to generals in the field, so they 
could study the theater of battle and consider tactics. These maps are called 
"sand tables" because they were originally created by generals thousands of 
years ago using actual trays of sand. Military leaders still do the same thing 
when they don't have a better map on hand (Figure 1.6). 

X-Men (2000). 

Learning Lessons from Science Fiction 


President Lyndon 
Johnson consulting 
a sand table of Khe 
Sanh during the 
Vietnam War. 

The main problems with modern 3D sand tables, while very accurate, are 
that they're expensive, static, somewhat delicate to transport, and useless if 
you guessed the wrong terrain. 

The animated pin board Douglas saw in X-Men solved a number of these 
problems all at once. Such a table could depict the topography of any 
location in the world, at any scale, at any time, and a general would ideally 
only ever need one. 

When he went back into work, he immediately wrote a request for proposal 
that referenced the scene in the film, so that military contractors would be 
inspired in the same way. One of the companies responding to the proposal, 
Xenovision, was awarded the development contract and, four years later, 
developed a working model: the Xenotran Mark II Dynamic Sand Table 
(Figure 1.7). 

The Xenotran Mark II 
Dynamic Sand Table, 
with its top raised. 


Chapter 1 

A still from a video 
showing the Xenotran 
Mark II Dynamic Sand 
Table with active 
topography and 
projected satellite 

The Mark II independently moves small metal rods that, together, create a 
new surface, much like what is implied in X-Men. Alone, this solution closely 
matches the technology implied in the film. While in development, though, 
the team took the concept even further. They covered the pins with a thin, 
white rubber sheet and vacuum-sealed it to create a smooth surface across 
the pins. Then, they projected imagery onto the surface from above, creating 
topography in full relief, with up-to-date satellite imagery and overlays of 
data (Figure 1.8). All of it can change over time, to create realistic, animated 
surfaces, depict tsunamis traveling across the sea, or even show landscapes 
shifting over geologic time. 

The main lesson from this story is that the technology might never have been 
developed if Douglas hadn't seen the film. 


Sci-fi, with its ability to present design fictions of speculative 
technologies with only narrative constraints, can do more than 
entertain us. It can inspire us with what's possible, what's ideal, 
and what would just be plain awesome. This book is meant to 
encourage you to look at sci-fi in the same way and come away 
inspired and ready to change the world. 

Let's Begin 

Now that we have outlined our constraints, explained our intentions, and 
gotten the coordinates from the navicomputer, let's make the jump to 
light speed. 

Learning Lessons from Science Fiction 




At First, Mechanical Controls Were Nowhere 16 

Then They Were Everywhere 17 
For a While, Mechanical Controls Started 

Disappearing 21 

Now They Coexist with Other Interfaces 24 

Mechanical Controls Are Used to Evoke Moods 26 

Mechanical Controls: Will We Come Full Circle? 27 

Science fiction is always rooted in the present, and it almost always 
reflects contemporary paradigms. This phenomenon is no more 
apparent than when looking at the common mechanical controls 
used to interact with devices— mechanical interfaces. As we'll see, buttons, 
knobs, and switches have been a mainstay of interface controls, both in 
reality and in sci-fi, since the early days of sci-fi, and they still show up in 
interfaces today, despite the sophisticated mechanical and virtual controls 
now available. Partly this is due to history and legacy: digital controls, such 
as touch screens, require sophisticated technology that only recently has 
become available. But mostly it's due to the fact that our hands are facile, 
and tactile and mechanical controls make fine use of these aspects of our 
fingers and bodies. (Our feet can control pedals, too, when our hands are 
busy, but we don't see a lot of this in sci-fi.) Let's head back to the beginning 
and see what role these mechanical controls were playing in 1902. 

At First, Mechanical Controls 
Were Nowhere 

In the first sci-fi film, Le voyage dans la lune, one detail that may be surprising 
to modern viewers is that it contains nothing that a modern audience would 
recognize as an interface. When the "astronomers" open the rocket door, they 
simply push on it — there's not even a handle (Figure 2.1). To launch the rocket, 
they load it, bullet style, into an oversized gun and shoot it at the moon. That 
there are no interfaces isn't really surprising, because this short movie is a 
vaudevillian comedy sketch put to film. But more to the point, when the film 
was released at the turn of the 20th century, very few interfaces existed in the 
modern sense. Audiences and filmmakers alike were working in an industrial 
age paradigm. The few controls that did exist in the world at this time were 
mechanical. People interacted with them using physical force, such as pulling 
a lever, pushing a button, or turning a knob. 


Le voyage dans la lune 

(1902, restoration 

of the hand-colored 



Chapter 2 

More Direct Than "Direct' 

Industrial age users experienced direct, mechanical feedback in the interfaces 
they used. For instance, if they pushed a lever forward, the machine would 
translate that to mechanical motion elsewhere. There was little abstraction 
between cause and effect. With the dawn of computation, the feedback 
between cause and effect became abstracted in the circuits of the machine. 
Pressing a button could result in any of a practically infinite number of 
responses, or it might not result in anything at all. This abstraction continued 
through the development of the DOS prompt. 

The development of the graphical user interface (GUI) returned some of the 
principles of the physical world to the experience of computing. For example, 
users could drag file icons representing data on a disk into a folder icon to move 
or copy it. Designers called the tight relationship between user actions and 
interface elements "direct manipulation," since it was more direct than typing 
commands via a text interface. Still, even with this physical metaphor, mechanical 
manipulations are much more direct than these "direct manipulations." 

Then They Were Everywhere 

In the 1920s and 1930s, as the developed world moved into the electric age, 
buttons, switches, and knobs made their way into industrial machinery and 
consumer goods that people used every day. As a result, these mechanical 
controls began to appear everywhere in sci-fi, too. In one example, the control 
panel from the Lower City in the 1927 dystopian film Metropolis shows an 
interface crowded with electric outputs and controls (Figure 2.2). As we 
continue to trace interfaces throughout this section, note the continued 
dominance of mechanical controls like momentary buttons, sliders, and knobs. 

Metropolis (1927). 

Mechanical Controls 


FIGURE 2.3a-c 

Buck Rogers, "Tragedy on Saturn" (c. 1939). 

World War I played a role in shaping the physical appearance of sci-fi 
interfaces as well, as servicemen brought their experiences with military 
technology back home as consumers, audiences, and sci-fi makers. In the 
1939 serial Buck Rogers, we see this in action. Buttons already inhabit the 
interface at this point, as in the "Tele-vi" wall viewer, controlled by just a 
few knobs, like televisions of the day (Figure 2.3a). When Captain Rankin 
and Professor Huer surmise that one of Killer Kane's ships they've detected 
is being flown by Buck, they want to contact the ship. Instead of invoking 
audio functions right there at the screen, they move to an adjacent "radio 
room" where they can hail him (Figure 2.3b, c). To modern audiences this 
seems silly. Why aren't these two capabilities located in the same spot? But 
the state of military technology at the time held that the radio room was a 
special place where this equipment was operated, even if it was set far apart 
from a periscope or other viewing device. 

Sci-fi has long built its spacefaring notions by extending seafaring 
metaphors. (The word astronaut literally means "star sailor.") By the 1940s 
and 1950s, sci-fi films like Forbidden Planet typically depicted its starship 
interfaces with large banks of mechanical controls of many types, such as 
those that sailors might have seen in the control rooms of great ships of 
World War II (Figure 2.4). 


Forbidden Planet (1956). 


Chapter 2 


As the examples in Metropolis and Buck Rogers show, new 
interfaces are most understandable when they build on what 
users (and audiences) already know. If an interface is too for- 
eign, it's easy for users to get lost trying to understand what 
the interface is or how it works. This is true of novice users and 
those who are not interested in technology for its own sake. It's 
also true for applications that are meant to be used intermit- 
tently or in a state of distraction. 

Make the interface easier to learn by providing familiar cues 
to what its elements are and how they fit together. This could 
mean building on current interface conventions or controls that 
map to the physical world. Metaphors can also be a bridge to 
this kind of learning as they help form analogies for users to 
make connections between things they already know and new 
interface elements that confront them. But take care, because 
holding too closely to a metaphor can become pointless skeuo- 
morphism 1 or confuse users when the interface's capabilities 
and metaphor diverge. 

Often, the mechanical controls of early sci-fi seemed disconnected from 
displays and neatly ordered by type in rows as in the image from Forbidden 
Planet. In some cases, like the 1951 film When Worlds Collide, production 
designers imagined putting the controls around the displays, where 
the user's actions and the system's results would be more connected. In 
Figure 2.5, the V and F knobs control the spaceship's trajectory, seen on the 
display as white points along the red and green lines. 

1 ^-1 — ^^ 


"'" 111 1 

oo : t ... ^ 



When Worlds Collide 


1 Objects that retain a decorative appearance from previous technological solutions despite no 
longer being required. 

Mechanical Controls 


In Buck Rogers, the two parts of the communication interface are in separate 
rooms. If Professor Huer wanted to tell Buck how to level his spaceship, 
the professor would have to run to the radio room to provide spoken 
instructions, or input, and back to the Tele-vi to check on Buck's progress, 
the output (see Figure 2.3b, c). Where the Buck Rogers scenario requires far 
too much work to be considered an efficient feedback loop, the navigation 
interface from When Worlds Collide is much tighter, with its controls 
abutting the output screen and fuel gauge, making much less work for its 
navigator. Imagine the disaster if the V and F controls were in the next room. 


Interaction designers call the cycle between input and output 
while optimizing toward some desired state a feedback loop. 
The faster and more fluid these loops are, the more a user 
can get into the flow of use and concentrate on managing the 
system to the desired state. Even when the controls in ques- 
tion are all on screen rather than mechanical, the more that a 
designer can do to tighten these loops, the more effective the 
user's interaction will be. 

This period also saw early sci-fi endeavors to depict the future with a 
dedicated realism. For example, Destination Moon made an earnest 
attempt to describe a trip to the moon with real science and plot, 19 years 
before the Apollo 11 mission would launch. Renowned science fiction 
author Robert Heinlein acted as contributor and technical advisor to the 
film. Unlike competing films of the era, which simply crammed as many 
buttons, switches, and knobs as possible onto the set to make them look 
sophisticated, Destination Moon portrayed a more serious and believable 
story through its constrained interfaces of more-considered controls and 
displays (Figure 2.6). These suggested a thoughtful reality behind them, even 
though at the time they were still entirely fictional. They were designed as if 
they could be real, not unlike a real prototype of a spaceship might be. 

Destination Moon 

20 Chapter 2 

Design for Dreaming 

By the 1950s, buttons, switches, and knobs were seen as a panacea to life's 
drudgeries. A delightful example of this appears in General Motors' 1956 
production of its annual touring auto show Motorama. It was one of the first 
examples of a corporation creating speculative fiction to promote its brand 
(and although not technically sci-fi, illustrative enough to bear mention here). 
It included the wonderfully melodramatic industrial film Design for Dreaming, 
in which a near-future housewife prepares a meal while dancing around her 
Frigidaire kitchen of the future (Figure 2.7). All of her once-dreary tasks, such 
as baking a cake, carrying dishes, and cleaning up, were accomplished simply 
by pushing buttons. 

Between the 1950s and 1980s, the trend for mechanical controls continued, 
despite a few new interface paradigms appearing. There were voice 
interfaces on robots, like Gort in The Day the Earth Stood Still (1951), and 
Twiki with Dr. Theopolis from the 1979 TV series Buck Rogers in the 25th 
Century, as well as artificial intelligences like the ship computers in Star Trek 
and 2001: A Space Odyssey There was also a gestural interface in The Day 
the Earth Stood Still. (See Chapters 5 and 6 on gestural and sonic interfaces 
for more on these topics.) These alternative interfaces were in the minority, 
though, until some budget constraints introduced a new paradigm. 

For a While, Mechanical Controls 
Started Disappearing 

When Star Trek: The Next Generation was green-lighted for production in the 
mid-1980s, the budget didn't allow for the same kind of jewel-like buttons 
as in the original series (Figure 2.8a). The money for so many buttons, 
individually installed and lit, simply wasn't available. Instead, production 
designer Michael Okuda and his staff devised an elegant and much less 

Mechanical Controls 


FIGURE 2.8a,b 

Star Trek: The Original Series (1968); Star Trek: The Next Generation (1987) 

LCARS interface. 

expensive solution — vast backlit panels of plastic film with simply printed 
graphics representing controls (Figure 2.8b). The result was thoughtfully 
futuristic as well as cost-effective, and the result inadvertently launched a 
new paradigm in interfaces that we see throughout the sci-fi genre today, 
in which controls exist only as a flat touch-screen surface. In Star Trek, 
this interface is known as LCARS (Library Computer Access and Retrieval 
System). Though the characters never use this term in the shows or films, it 
has appeared in some of the on-screen interfaces. (We'll discuss it in detail 
in the next chapter as a case study in visual design.) 

Setting the stage for Okuda's solution were new, experimental interfaces 
that had been developing between the first and second Star Trek series. One 
example is the Aesthedes computer, a graphics workstation produced by 
the Dutch company Claessens in 1982 (Figure 2.9). The computer's functions 
were arrayed across a seamless tabletop surface, with each function given 
a separate button, arranged in logical groups. This made it wide enough 
to be a literal desktop of controls. Like the LCARS interface, these buttons 
had almost no depth. Instead, they were part of a seamless membrane that 
stretched over the entire surface, printed with labels and borders, with 
simple contact sensors positioned underneath. They were a transitional 
technology, still mechanical but very different than the kinds of buttons seen 
and used before. They represented a half step toward interfaces like LCARS, 
but, unlike LCARS, they weren't changeable, since they weren't also a display. 
This style of mechanical interface, on the border between mechanical and 
virtual, didn't catch on, but during this time of experimentation, it was an 
option very much like what was explored in Star Trek: The Next Generation. 


Chapter 2 

Aesthedes computer 
(c. 1982). 

Less Is More and Less 

Are fewer controls with more modes better than more controls with fewer modes? In the 
Star Trek and Aesthedes examples (see Figures 2.8 and 2.9), each button controls only one 
function (as best as we can tell from studying the use of the fictional Star Trek interfaces). 
Contrast that with contemporary computer systems in which controls perform multiple 
duties. The A key on your keyboard, for example, can mean the letter a or Select All or a, 
and so on. 

These examples raise similar questions with displays. Are fewer displays with more modes 
better than more displays with fewer modes? Star Trek shows different kinds of information 
and different views of the same information on a few screens, but the Aesthedes had 
multiple screens dedicated to different types of information. 

The tension between ease-of-use and control is a central element in interaction design. 
It isn't clear whether there is a one-size-fits-all approach, but one best practice is to let 
the user's experience with the interface be a first determining factor. For users who are 
untrained or use an interface only occasionally, you can ease their learning curve and 
reduce the burden on their short-term memory by providing more controls with clear 
labeling and fewer modes. For expert users, you can increase their speed and efficiency by 
providing fewer controls with easily accessible and memorable modes. 

These issues are compounded by the design constraint of cost. For example, the LCARS 
interface with its large, backlit control panels was created for Star Trek: The Next 
Generation because the budget didn't allow for banks of mechanical switches as in the 
original Star Trek series. And the Aesthedes couldn't compete with the much less expensive 
IBM PC XT with its fewer component parts. Designers working on the industrial design of 
their products need to balance this constraint with those of usability and learnability. 

Mechanical Controls 


Now They Coexist with Other Interfaces 

We see that even in modern sci-fi with advanced digital controls, mechanical 
controls are still present. In the reboot of Star Trek the interfaces on the 
Enterprise use a blend of touch-screen surfaces and mechanical controls. The 
throttle for the helm is mechanical and familiar for the audience, who has 
experienced or seen similar controls on ships and airplanes (Figure 2.10). 

One of the benefits of mechanical controls is that, unlike touch-screen 
controls, they can be well-designed for our entire hands rather than just the 
fingertips, offering ergonomic shapes and rich haptic feedback. Additionally, 
mechanical controls can take advantage of a user's finer motor control 
and offer industrial design that telegraphs to the user how it can be used. 
For example, the shape of a button or knob might better communicate 
optimal position or the amount of force to be used. The diameter of a knob 
can make fine control easier or more difficult, depending on the exertion 
needed for fingers or hands to move it. This not only increases control and 
comfort, making some actions easier but also communicating function 
through the physical form, itself, the property that interaction designers call 


Mechanical controls are more appropriate when fine motor 
control is needed. It's not that screen controls can't accept fine 
movement, but, as many users find with their trackpads, touch 
interfaces are often so sensitive to movement that holding a 
specific position is difficult. For example, taking your fingers off 
of a knob doesn't change its position, but it can change a sen- 
sitive touch-screen control, even if that isn't the intent (as can 
touching it again). Users can "have their finger on the button" 
without actually depressing it, but on-screen buttons can be 
unintentionally or prematurely activated in this way. 

FIGURE 2.10 
Star Trek (2009). 

24 Chapter 2 


Even with new advances in natural gesture technologies, such 
as Microsoft's Kinect, there is comfort and ease in mechani- 
cal controls for some operations. Take typing for example. It's 
terrible with the Kinect, OK with a controller, and much better 
with a physical keyboard that has been optimized for this 
purpose. Voice control may make even keyboards obsolete, 
but they have their own limitations. 

Screen-based controls can mimic some of what mechanical 
controls have always offered, like a satisfying click when turned 
to the desired position, or alignment between several buttons 
to indicate common settings. And screen-based controls can 
do many more things than mechanical controls, such as incor- 
porating animation, appearing only when needed, or changing 
entirely based on context. Finding the right control for the job 
is why even the most advanced smartphones still have a few 
mechanical buttons for controls like Volume, Home, and Power. 

Just as in real design and engineering, the presence of mechanical controls 
followed trends pertaining to material costs and scarcity. At one time, buttons 
and other components were relatively inexpensive due to their materials 
and manufacturing processes, so they abounded in both real and fictional 
interfaces. When the sheer number of controls and their expense (both for 
materials, installation, and maintenance) rose, however, they began to be 
used more sparingly. We see this today as touch-screen interfaces are able to 
incorporate many functions with no added cost, so many mechanical controls 
are disappearing. In addition, too many undifferentiated buttons can breed 
confusion, and overload for audiences and users alike. 

For example, we see this in the lineage of Star Trek interfaces. The original 
series used many, individually lit buttons, often positioned in rows circling 
the user (which is more complicated than rectilinear rows). But when 
the production budget available for the next series, Star Trek: The Next 
Generation, didn't go as far, that was no longer an option. Instead, the 
mechanical buttons disappeared almost entirely from control interfaces, 
in lieu of flat-panel touch screens. This persisted throughout the following 
TV series and many of the films, until the latest film, Star Trek, created 
a rebooted aesthetic based on a complement of mechanical and touch- 
screen interfaces. 

Mechanical Controls 25 

FIGURE 2.11 

Star Trek: Insurrection 


Be warned, though, that hybrid controls can still seem out of place or even 
laughable. A funny example comes from the climactic sequence at the end 
otStar Trek: Insurrection. After an entire film otStar Trek's signature touch- 
screen (LCARS) interfaces for controlling nearly everything that happens on 
the ship, Commander Riker calls for the "manual weapons interface," and a 
1990s-era joystick pops up from a console designed specifically for this one 
purpose (Figure 2.11). It isn't that this is a worse interface for this use — indeed, 
it may actually be better. The joke is that everything in the film leading up 
to this, including flying the ship and shooting weapons, never used such an 
interface. If it was a useful and even better weapons control, why wouldn't it 
have been the used in battles before this moment in the story? 



Mechanical controls are better for some uses, though they can't 
as easily serve multiple functions. Nonmechanical controls, like 
touch-screen buttons, are easier to change into other controls 
but don't offer the same kind of haptic feedback, making them 
impossible to identify without looking at them and creating 
questions about whether they've been actuated. Design inter- 
faces with an appropriate combination that best fits the various 
uses and characteristics. 

Mechanical Controls Are 
Used to Evoke Moods 

Sci-fi can use mechanical controls to evoke moods. For example, the use of 
almost exclusively mechanical controls can help to establish a steampunk 
or alternate-history feeling. In the film Brazil, a patchwork of elements 
from many eras grafted onto each other references the hazards of an overly 
authoritarian, anonymous, and hyper-administrative future (Figure 2.12). 
Any comfort that the audience might find in familiar controls only 
underscores the nightmare of a dystopian information age. 

26 Chapter 2 

FIGURE 2.12 
Brazil (1985). 


The interface seen in Brazil doesn't look ready for use (see 
Figure 2.12). It looks haphazard, thrown together, and proto- 
typical. It looks like it would be hard to figure out and easy to 
break. Director Terry Gilliam was imparting a sense of dystopia 
throughout the film, so we can take this computer interface to 
underscore what should not be done for real-world systems. 
System interfaces that are instead cohesive and look whole, 
complete, and considered inspire confidence in users and 
reduce anxiety. 

Mechanical Controls: Will 
We Come Full Circle? 

Not only do mechanical controls and virtual ones coexist today, both in sci-fi 
and real-world interfaces, but they are likely to in the future as well. This is 
because each offers benefits that are important to our use of systems. Our 
hands still have physical characteristics that make touch, hand position, and 
fine motor control a better interface for those tasks that benefit from them. 
In addition, many complicated interfaces now must fit into small displays, 
such as on tablets and phones. As such, they must change elements within 
the screens in order to show all controls in these small spaces. This means 
that mechanical buttons can't be used for all controls since their type and 
orientation may not fit the next screen of controls. This coexistence isn't 
surprising, but it will require interface designers to understand and determine 
when each serves the best purpose and how best to put them all together. 

Mechanical Controls 


The recent popularization of gestural interfaces and voice controls hint at 
a future where computer systems watch and listen to us for their inputs 
instead of our poking or prodding at them. Though these technologies are 
crude and inaccurate at the moment, as they mature and become more 
ubiquitous and precise, will the need for mechanical controls disappear? 
Will we return once again to a world like the magical one envisioned by the 
filmmaker Georges Melies in Le voyage dans la lune at the dawn of cinema, 
when to open a spaceship we would just push at its door? 

28 Chapter 2 


Visual Interfaces 

What Counts? 


Text-Based Interfaces 


Graphical User Interfaces 


Visual Style 


Visual Interfaces Paint Our Most Detailed 

Pictures of the Future 


Jurassic Park (1993). 

Most fans of the film Jurassic Park remember the tense scene in 
which two fearsome velociraptors chase the park's remaining 
visitors into a computer lab, and while the two adults desperately 
try to barricade the door, the two children rush to a computer terminal. 
The young girl, Lex, looks at the screen and proclaims, "It's a UNIX system. 
I know this." It's a memorable geek moment (Figure 3.1). 

Lex then searches for the lab building controls so she can activate the door 
locks to keep the velociraptors out. (Let's not inquire, for the moment, why the 
doors themselves don't have lock controls on or near them.) As she uses the 
workstation, we see a spatial user interface at work— an actual Silicon Graphics 
product from the time the film was made, known as 3D File System Navigator. 

As the camera cuts back and forth between the kids at the computer and the 
adults blocking the door, the filmmakers are careful to show Lex's hand as 
she uses the mouse. Although it wouldn't be necessary today, back in 1993 
many audience members would not have been familiar with a graphical user 
interface (GUI) with a windowing system and mouse. 

There's an odd moment, however, as Lex navigates spatially to the block 
that represents the lab so she can click on it to access the controls. At first, 
it's a small block on the horizon, and it only shifts forward slowly as the 3D 
map is redrawn (Figure 3.2). Once the desired block is in the center of the 
screen, Lex clicks on it to access its controls. The music swells to increase the 
tension — will she make it in time before the dinosaurs break in? 

But if she can see the block on the screen when it's on the "horizon," can't she 
simply click on it and activate the locks immediately? Why must she (and the 
audience) wait until it comes closer to the center of the screen and becomes 
a bigger target? 

30 Chapter 3 

Jurassic Park (1993). 

The answer is that it doesn't — or shouldn't — except that the director and 
writer needed to create tension in the scene, and this was a novel way to do 
so. The film sacrifices credibility in order to generate this tension, because, 
in real life, an interface like this should be quick and easy to use. 

A spatial metaphor is a sensible way to organize locational data. It can 
even be useful in situations in which the data isn't geographic (think of a 
spatial map of the body or of an automobile). While the use of a 3D spatial 
file system in Jurassic Park highlights an example of how great ideas in 
interfaces can become misused in films and television to serve the narrative, 
the more important point is that the novel visual style of the interface was 
purposely selected for its futuristic look. The director and designers weren't 
content with a standard visual interface of the time, such as a floating 
window with buttons arranged inside. That would not believably establish 
that the organization was technologically advanced enough to grow 
dinosaurs from scraps of ancient DNA. 


Interaction designers are used to thinking in terms of goals and 
users navigating complex systems to achieve those goals. The 
appearance of a system can be relegated to secondary impor- 
tance. But the visual design shapes the user's impression of the 
system— how it compares to competing systems, its desirability, 
and even how capable and usable it is. 1 Designers ignore ap- 
pearance at their— and their users'— peril. 

1 Kurosu, M., & Kashimura, K. (1995). Apparent usability vs. inherent usability: Experimental 
analysis on the determinants of the apparent usability. In J. Miller, I. R. Katz, R. L. Mack & 1. 
Marks (Eds.), CHI '95 conference companion on human factors in computing systems: Mosaic of 
creativity {pp. 292-293). New York: ACM. 

Visual Interfaces 


What Counts? 

Television and film are visual and auditory media. Cataloging every visual 
element or style throughout sci-fi would take much more space than we've 
got, so, for brevity, we have limited the elements to the use of graphics and 
text to convey information and controls. 

Text-Based Interfaces 

The visual design of computer interfaces began simply: with text on screens 
and little else. 

Command-Line Interfaces 

Early computer interfaces displayed input prompts, commands, responses, 
and system status as text. The earliest stored-program computers, such as 
ENIAC, printed their output on paper punch cards for the programmers 
to read. Later, time-share systems, which used remote teletype terminals 
to communicate, typed system status and commands on text-based paper 
rolls. When cathode ray tubes (CRTs) became more feasible, screens became 
the primary display technology. All of these were command-line interfaces 
(CLIs), and CRTs signaled the rapid expansion of computers into all aspects 
of work. Hollywood wouldn't catch up to show CLIs on its screens for a few 
decades, by which time the style of these interfaces was firmly established 
in the real world. 

Beginning in the 1950s, the typography of command-line interfaces 
was bitmapped and almost always fixed-width capital letters. Although 
as of the early 1960s ASCII encoding included lowercase letters, many 
implementations on popular platforms of the time, like CDC (Control Data 
Corporation) and DEC (Digital Equipment Corporation), didn't allow for the 
use of lowercase characters. These screens resembled the output of cheap 
typewriters with a stuck shift key more than the professional typesetting 
used in publishing at the time. 

Though all-text output had been seen in some prior films like Logans 
Run (1976), the first mainstream CLI in the survey appeared in the 1979 
movie Alien, as Captain Dallas communicates to the ship's control system, 
named Mother. Mother's menace is underscored by her inhuman quiet and 
emotionlessness (Figure 3.3). This interface embodies the look of computer 
systems three decades old at the time, but it established a style that could be 
seen for decades to come. 

32 Chapter 3 

Alien (1979). 


PART TWO; 17:81:35 


FIGURE 3.4a,b 

War Games (1983); Brainstorm (1983). 

Since the advent of advanced GUIs with more refined typographic details 
that better resembled the typography in print, these crude bitmapped and 
fixed-pitch typefaces (and, to some degree, command-line interfaces in 
general) are used to denote either an older system or a character's having 
deep access to a system. Gone, too, are the common use of all capital letters 
and the limited number of glyphs available in early computer typefaces. Now 
this style serves to bring to mind a very particular date and time, as seen in 
the examples from War Games and Brainstorm (Figure 3.4). 



Nothing ages an interface like all capital letters, lots of num- 
bers, little punctuation, and no accents. In addition, the more 
bitmapped the type is, the older it looks, reflecting a time when 
screen resolutions were low and computer typefaces were 
pixel-based instead of constructed from outlines. 

Visual Interfaces 



Because people first read word shape rather than the individual 
letters in each word, fixed-pitch typefaces and, even worse, 
text in all capitals are more difficult to read. The variations in 
letter rhythm and shape make words distinctive and help us to 
identify them quickly. But it's still a part of our collective visual 
language to see this as less sophisticated typography and iden- 
tify it as more serious and aimed at high-functioning experts, 
even if it's much slower to read. 

Once established, command-line interfaces were the dominant interface 
for nearly 50 years and still haven't gone away entirely, in part because 
this is how most programmers learned to program, and in part because 
it's often faster and more efficient to code in text-based interfaces — since 
code is text. Although 2D GUIs have risen in prevalence since the 1980s, 
we still see command-line interfaces in sci-fi. For the most part, directors, 
writers, and designers today use command-line interfaces when they want 
to communicate one of two things: that a system is older, or that a user 
has sophisticated computer skills. Sometimes a character with technical 
expertise will drop into a command-line interface in a window of a more 
sophisticated graphical interface for just this reason. One such example 
occurs in The Matrix Reloaded, when Trinity breaks into the power utility's 
control center. Despite the many more-sophisticated graphical interfaces 
available, in order to defeat the security system and shut down the city's 
power grid, she drops into a command-line interface to conduct her 
sabotage (Figure 3.5). 

jj *■*£&' ffii**' 


FJi?»J3 fjqp| _ ^t^4£*§? i * |l 


FIGURE 3.5a-c 

The Matrix Reloaded (2003). 


Chapter 3 

<s cr i pt s r c= 1 Vu i/s cri pt/rm- bos e , j s 7t=07101Z- l^WW* </s cri pt> 

<s cri pt s r c= Vui/s cri pt/s ect ion- homepogeZ . j s 7 t=87101Z- 1400 "> </s c r i p t> 
<script src= ,- http ://use . type kit . com/ntvlf pz t js"x/script> 
<s cri pt> try {Type kit . loadQ ;} catenae )fj</s cri pt> 

<link rel="Shortcut Icon" type="irriaqe/ico" href=' r http:/Vwvw. ■" 

<linik rel= 1p alternate" type="application/rss+xml" title= ,, RSS 11 href="http://feeds t rosenf 

<body id="rosenfeldmedia-com" class=" sect ion- home layout-content with-sidebar" data-curren 
<div irt="accessibility"> 

<a href='"#mnin-rinv'">Skip to Navigation</a> I 

<a href= 1h #CGntent-area">Skip to Content</a> 
<div id= 1 * container "xdiv closs="c , > 


<div class=" account -bar style -paper "xdiv class='"wrap'"> 


<li class= "total "><a href= 1 Vcheckout/cart.php'">Your cart is empty </a></li> 
<l ixa hr ef = 1 YcheckoLit/logi n . php"> Logi n</a> </l i> 
<li class="last"><a href='Ycontact/'\>Contact</a></li> 
</divx/div>< ! - - /account-bar --> 
<div class=" banner %<div class= 1 Wap% 
<div clas s=" logo-type 1 '^ 

<a href ="/"><spanj>Rosenf eld Media</spanx/a> 


Adobe Dreamweaver CS6 code interface. 

Despite the technical sophistication they convey, command-lines interfaces 
can be difficult to parse and scan to find a specific line or command, 
especially when everything looks the same. This is where real systems have 
excelled past fictional ones. Coding interfaces such as Adobe's Dreamweaver 
apply color to HTML code to make it easier to parse (Figure 3.6). Tags are 
blue. Links are green. Functions are purple. Different coding environments 
use different color schemes, but it's a rare system that doesn't help break 
up the "wall of code" with color. We only see hints of this in sci-fi, probably 
because the interfaces are there to embody plot, rather than be used. 


These kinds of "Wow, they're good!" moments work because 
the interface looks to the novice overfull, undifferentiated, 
and complex, while the hacker works through it at breakneck 
speed. Real-world experts enjoy the same social cache when 
they have mastered a tool that appears dauntingly complex to 
a novice. The expert is rewarded with respect in these mo- 
ments, and the need for the services of the expert is reinforced. 
To ensure that these moments can happen, designers need to 
include ways for the expert's mastery to be seen and appreci- 
ated—without being too understandable. 

Visual Interfaces 35 

Graphical User Interfaces 

Interfaces that move beyond the command line are considered graphic 
user interfaces, or GUI. These include those with WIMP interface elements 
(windows, icons, menus, pointing devices) as well as those with shadows, 
refined typography, layering, and graphic controls such as buttons. Because 
there are so many GUIs to consider, this section looks at these interfaces 
by component. 


We reviewed the main typefaces for each property in the survey. This wasn't 
always easy, as typefaces aren't always identifiable, even with the help of 
resources like, and it's often tough to identify which might be 
the main typefaces of several used. With those caveats, here's what we found. 

Typographers may not be surprised to learn that sans serif is the 
overwhelming typeface choice for sci-fi, with serifed typefaces appearing in 
only a handful of interfaces. In the current survey of movies and TV shows, 
only seven serif typefaces were found: Alien (see Figure 3.3 above), Blade 
Runner, Galaxy Quest, Gattaca, The Matrix, Men in Black, and Star Trek: 
The Original Series (Figure 3.7). Note that in Star Trek and Blade Runner, all 
interfaces other than the one pictured here use sans serif type. Only Gattaca 
uses serif typefaces throughout. With that in mind, the ratio between sans 
serif and serif typefaces might be closer to 100:1. 

FIGURE 3.7a-e 

Star Trek: The Original 

Series (1968); Gattaca (1997); 

Blade Runner (1982); Until the 

End of the World (1991); 



Chapter 3 

4% Misc Serif 

4% Misc Sans Serif 

Helvetica 32% 2SCourier 

EurOStile 28% J 3% Chicago 

4% OCR B 
17. OCR A 
8% Futura 

8% "l-SEGPIEriT 
The typefaces seen in the survey are overwhelmingly sans serif. 

When we could identify specific typefaces, the majority were either 
Helvetica (or a derivative like Arial) or modular typefaces like Eurostile 
or Microgramma. The next third divides between Futura, OCR A, and 
LED-type faces. OCR B, Chicago, and Courier appeared in more than one 
property. Swiss 911 Ultra Compressed, the typeface used in the Star Trek 
LCARS interface, is underrepresented here because we counted it as a single 
property. Based on screen time, this one typeface might well eclipse all of 
the others. After counting typefaces per property, we get the results shown 
in Figure 3.8. 


Given the propensity of sci-fi to use sans serif typefaces, de- 
signers working on projects meant to have a futuristic or sci-fi 
feel choose sans serif typefaces. If designers wish to build di- 
rectly on the cache of sci-fi, Helvetica and Eurostile are strong 
candidates that viewers are accustomed to seeing. 

Typography in interfaces largely conforms to standards of the real-world 
systems being mimicked, with one primary exception. Cinematic interfaces 
must be read by audiences much more quickly than their real-world 
equivalents, and the visual hierarchy often becomes much more exaggerated 
to draw attention to the aspects of the interface that are important to the 
plot (Figure 3.9). In a few cases, this makes sense in the real world if the 
system is truly critical or built for true novices, but in systems built for 
expert users we would not expect the overly large text labels. 

Visual Interfaces 37 

FIGURE 3.9a-i 

Logan's Run (1976); Terminator 2: 
Judgment Day (1991); Independence 
Day (1996); Starship Troopers (1997); 
The Fifth Element (1997); Mission 
to Mars (2000); X2 (2003); The 
Incredibles (2004); Eagle Eye (2008). 



As desktop publishing and GUIs became more sophisticated in the early 
1990s, more sophisticated typography appears in the survey, as well. 

Typography on paper has been evolving for hundreds of years and has been 
optimized for its high-resolution medium. While screen-based typography 
has some of its own, unique requirements, as screen resolutions have 
increased steadily since the 1990s, the principles of print-based typography 
have become both more and more applicable to screens and possible to 
implement. For example, compare the displays in The Island, which use a 


Chapter 3 

FIGURE 3.10 
The Island (2005). 

great deal of text, with the command-line interfaces in the prior section. 
The Island interfaces are richer and more legible partly because they use 
a variety of type sizes and faces to distinguish information (Figure 3.10). 
Most importantly, first-read information is larger and uses both upper and 
lowercase letters. Repetitive, second-read, and less important information 
uses only uppercase letters and smaller sizes. Third-read information is 
colored to have less contrast so that it stands out less, and fourth-read 
data— the least important— is very small. Sci-fi designers were maximizing 
the screen technology available to them and reminding us how far display 
technology has come. 


The graphic style of early GUIs was partly a result of the con- 
straints of the technology. Today, high resolutions and detailed 
graphic control is the norm. Without losing the new principles 
learned from on-screen design that deal with time, motion, 
and mode, designers can reincorporate the best practices from 
print that, for a time, had been disregarded as obsolete. 

Though we don't have enough room to get into all of them here, 
a few obvious ones jump to mind: use strong visual hierarchies 
to draw the eye, and use mixed cases as well as graphically cor- 
rect punctuation, diacritical marks, and ligatures. These prin- 
ciples can make text on screen much more legible and beautiful. 

Visual Interfaces 



The most prominent visual aspect of speculative technology is that it glows. 
From lightsabers to blasters, holograms to teleporters, most sci-fi technology 
emits light. It is the most common tag in our tag cloud from Chapter 1 (see 
Figure 1.3), and any casual overview of the survey shows its ubiquity. 

This effect includes on-screen elements as well. Type and other graphic 
elements, like lines in a map or diagram, are often a bright color on a dark 
background. Frequently a blur around these elements is added to enhance 
the glow effect. To push the technological aspects of the interface, diagrams 
and images are often rendered as wireframes instead of solid or patterned 
fills, creating more opportunities for high-contrast glowing (Figure 3.11). 


Why does sci-fi glow? We suspect it's because things of power 
in the natural world glow: lightning, the sun, and fire. Other 
heavenly bodies glow as well— stars, planets, and the moon 
(especially against a black background)— and have been long 
associated with the otherworldly. Additionally, living things that 
glow captivate us: fireflies, glowworms, mushrooms, and fish in 
the deep seas. It's worth noting that while most of real-world 
technology glows, a lot of it doesn't, so its ubiquity in sci-fi 
tells us that audiences and sci-fi makers consider it a crucial 
visual aspect. 

Regardless of the reasons, designers should be aware of this 
principle. If you want your interface or new technology to seem 
futuristic, it's got to glow. 

FIGURE 3.11a-c 

Star Trek (2009); Defying Gravity 

(2009); Avatar (2009). 


Chapter 3 

L r to ,9 ' 3 



L.. 1979 . 

1980 1982 1983 1 1986 1989 

^ — ^— i ^ -J L- 










[ 2009 



FIGURE 3.12 

The colors of sci-fi from the Make It So database show the strong tendency 

toward blue. 


The histograms in Figure 3.12 were made by selecting representative images 
for each screen-based interface in the survey, filtering out noninterface 
elements in the scene, aggregating them into a single image, and running a 
Photoshop analysis on the result. To create the color chips for each year, the 
same aggregate image was reduced to a single pixel, and its saturation was 
adjusted to 100 percent. 2 Though interesting, it should be taken with a grain 

2 Almost everyone who saw this graphic in development immediately asked what happened in 
1991. See our discussion of the cyborg vision from Terminator 2: Judgment Day in Chapter 8. 

Visual Interfaces 


FIGURE 3.13a-g 
Galaxy Quest (1999); Battlestar 
Galactica (2004); Supernova (2000); 
Fantastic Four (2005); Tfte Island 
(2005); Tfte Hitchhiker's Guide to the 
Galaxy (2005); Iron Man (2008). 

of salt, because there are a number of problems in regarding these results 
as scientific, both in content and process. Despite these caveats, one of the 
things clearly shown in the chart is that sci-fi interfaces are mostly blue 
(Figure 3.13). 


Why are sci-fi screens mostly blue? Mark Coleran, a noted pro- 
duction designer on many films, tells us that it's partly a techni- 
cal reason: tungsten lights are the most common ones used on 
sets, and they're very warm in color. As filmmakers compensate 
for this in post-processing, blue colors are affected the least. 
Maintaining the vibrancy of other colors is tricky at best. Even 
if blue colors shift across this process, it's hard to notice be- 
cause the human eye is least sensitive to the blue-yellow color 
axis. Trying to compensate for this color shift in the actual 
screen designs on set can result in garish interfaces that upset 
directors and actors, so it's often easier to just stick to blue. 


Chapter 3 

Is there also a psychological reason? It could just be that blue 
is the most popular color worldwide, as polls by companies 
like AkzoNobel 3 and Cheskin 4 seem to verify time and again. It 
could be that interfaces are places for work, and blue is often 
associated with coolness and calm. It could also be that blue 
is comparatively rarer in nature than other colors (with the 
exception of the sky), and it underscores the technology-ness 
of the technology. Or, it might be the opposite, since everyone 
sees the sky at some point no matter what climate they live in. 
Blue is often the color most associated with business, and it 
was often used in some of the earliest screen interfaces in the 
real world (as in the blue screen of death). 

Regardless of the reason, as with typeface and glow, designers 
should be aware of this trend. If you're looking to establish a 
futuristic interface, shades of blue are an easy bet. 

The majority of screen interfaces in sci-fi adhere to the conventions of blue 
glow, but the exceptions tend to be found in the most unique interfaces. The 
second most common interface color seen in the survey is red (Figure 3.14). 


FIGURE 3.14a-f 

The Fifth Element (1997); Red Planet 
(2000); Star Trek: Nemesis (2002); 
The Hitchhiker's Guide to the Galaxy 
(2005); Iron Man 2 (2010). 

3 AkzoNobel. (2012). Color futures 12. Retrieved from 

4 Cheskin, MSI-ITM & CMCD Visual Symbols Library. (2004). Global market bias: Part 1. Color: 
A series of studies on visual and brand language around the world. Retrieved from report-2004_ 


Visual Interfaces 



As common as blues are as a baseline for interfaces, red is 
common as an alert to danger, errors, or failure— including 
death. This reflects the largely consistent use of red in Western 
countries for stop signs and warnings of all types, so it's best 
not to attempt to counter this learned association and try to 
communicate danger in other ways, unless you're prepared for 
user error and lots of user retraining. 

Green is the third most common interface color in the survey. It is the color 
of hackers who work in command-line "green screens," directly reflecting 
the two decades that monochrome cathode ray tube (CRT) displays were 
the main ways to interact with computers. Wireframe 3D shapes and radar 
are almost always shown in green. Occasionally green is a contrast to 
red to clearly distinguish safe and dangerous states such as "locked" and 
"unlocked" (Figure 3.15). 

In Transformers, a US military interface adheres to the glow lesson in the 
section above but uses green as the predominant color, instead of blue 
(Figure 3.16). The interface still looks technological but also slightly less 
typical than it would if it used blue. 

FIGURE 3.15a-d 

Flight of the Navigator (1986); Matrix Reloaded (2003); 

Battlestar Galactica (2004); Firefly, "Safe" (Episode 5, 2002). 


Chapter 3 

FIGURE 3.16 
Transformers (2007). 

FIGURE 3.17a-d 

Blade Runner (1982); Star Wars Episode I: The Phantom Menace (1999); 

2001: A Space Odyssey (1968); Mission to Mars (2000). 

Yellow and orange often serve as attention-directing highlights, or for 
caution messages that are distinct from more severe warnings (Figure 3.17). 

Purple is the rarest interface color seen in the survey, possibly because 
the color has been technically hard to reproduce in the media of film and 
television. No trends of use can be observed in the samples from the survey. 

To differentiate the interfaces in the Star Trek prequel, Enterprise, from 
those later in the timeline, the designers used a predominantly gray color 
scheme to make the interfaces feel much closer to our own times than the 
more advanced ones further in the future. This effect is aided by rectangular 
windows and simple geometric buttons on these screens (Figure 3.18). 

Visual Interfaces 


FIGURE 3.18a,b 

Comparing interfaces in Enterprise (2001) and Star Trek: Deep Space Nine 

(1993), separated by 200 years in the property's timeline. 


Using a theme of gray elements on screen often makes an inter- 
face look less sophisticated and reminiscent of early-generation 
GUIs, before rich color screens were affordable and widespread. 

Original Uses of Color 

Blue and glow are common, but we also see examples of interfaces that don't 
use these conventions. 

In The Matrix Reloaded, we see one example of an all black-and-white 
interface (Figure 3.19). The Zion city control room exists in an all-white 
virtual space. The operations controllers there work with a black-line, 3D 
virtual touch interface arrayed in front of them. To make the "virtualness" 
even more apparent, these controllers are dressed all in spotless white as 
well. The overall effect is techy without resorting to typical conventions. This 
control "room" is part of the matrix and not a real space, so the convention of 
seamless white is used to indicate that everything in it, including the people, 
are virtual. The gray touch interface, as well as its translucency overlaid onto 
the scene, only reinforces this distinction from reality. 

One interface from the Torchwood TV series features a monitoring interface 
that uses red and pink highlights supported by other highly saturated colors 
(Figure 3.20). 

In Star Wars Episode I, the pod-race interface used by Anakin Sky walker 
glows, but with a predominantly orange interface with green and purplish- 
pink highlights. Included is some mostly yellow "filler" text in an alien 
language (Figure 3.21). 


Chapter 3 

FIGURE 3.19 

The Matrix Reloaded (2003). 

FIGURE 3.20 
Torchwood (2009). 

FIGURE 3.21 

Star Wars Episode I: The Phantom Menace (1999). 

Visual Interfaces 


r i 

FIGURE 3.22 
Space: 1999 (1975). 

A last example that goes against the blue and glow conventions comes from 
Space: 1999, in which a display within the moon base uses several bright, 
graphic colors with white lines and shapes to delineate measurements and 
white circles to track data (Figure 3.22). The overall effect looks nothing like 
the typical screen interfaces we see. 


Bucking convention is often a successful path to creating a 
unique and memorable impression. Use uncommon colors 
or color combinations in the interface that don't stick to the 
too-common blue and glow conventions to differentiate your 
interface from more typical ones. 

We must note an exception to these color trends. The LCARS interface, with 
its highly consistent palette, uses lots of yellows, oranges, blues, and purples 
throughout (see Figure 3.18b). This choice of using uncommon colors, 
added to the rest of the LCARS standards, makes a unique, memorable, and 
extensible interface. 

Color Coding 

Color coding is used in the larger franchises to help audiences distinguish 
the technology of the different civilizations. For example, in the later 
Star Trek series, starting with Star Trek: The Next Generation (1987-94), 
Borg technology is green (Figure 3.23a), Starfleet's is the LCARS palette 
(Figure 3.23b), Klingon technology is red (Figure 3.23c), and so on. Even 
the transporter streams of these different cultures have differentiated 
tints and appearance so that audiences can tell who is beaming in or out. 


Chapter 3 

When we see a green glow on a ship, we know it's Borg, even with little other 
information to go on. Similarly, when devices or ships appear on screen for 
the first time, their origins are often conveyed through color. 

As interface designers, we also can use this power of differentiation and 
identity. Of course, color doesn't need to convey this information alone. 
Other visual elements can assist, as well as the shape of the display screen, 
discussed below. 


When information is presented statically, designers have many 
tools at their disposal to suggest structure: visual hierarchy, 
grouping, lines, and so on. These help users understand parts 
of the interface, their relative importance, and the relationships 
between them. When information is distributed across time, 
these visual aids are less effective. In these cases, use color to 
reinforce related types of information. This relies on the judi- 
cious assignment of color to category. To increase this effect, 
combine color with other visual cues, such as typography, 
shape, repetition, texture, and differentiated motion. 

FIGURE 3.23a-c 

Star Trek: The Next Generation and 

Star Trek: Voyager (1986-2002). 



B . ' 





fr % ^B ' 

i^ , 

Mf m 



K^L\ 1 

Visual Interfaces 


Display Shape 

Nonrectangular computer screens are an entirely sci-fi concept, because 
OLED or LCD displays that stray from rectangular shapes still exist only 
in the lab. (Radar screens and oscilloscopes had circular screens, but these 
aren't GUIs.) The instances of nonrectangular screens we see throughout the 
survey helps make the interfaces look both futuristic and alien (Figure 3.24). 


Because all screen technologies in use today are rectangular, 
masking the full display or using graphics to create a nonrect- 
angular effect can make an interface appear more advanced. 

FIGURE 3.24a-e 
Star Wars Episode I: The Phantom 
Menace (1999); Men in Black II (2002); 
Firefly, "The Message" (Episode 12, 
2002); Star Wars Episode IV: A New 
Hope (1977); Doctor Who, "Rose" 
(Season 1, Episode 1, 2005). 


Chapter 3 


ff* wk — s 

FIGURE 3.25a,b 

Things to Come (1936, colorized version). 

Layers and Transparency 

Transparency has been a part of sci-fi displays almost from the beginning, 
mostly as physical, transparent screens. 

Transparent Displays 

Though these may seem like a contemporary idea, they are seen as early 
as 1936 in the film Things to Come, an adaptation of a Jules Verne novel in 
which John Cabell, a leader of a technocratic city of engineers, activates a 
completely clear screen to show his granddaughter the history of the city 
and how it grew out of the ashes of a decades-long world war. The video 
and the images themselves are somewhat translucent when viewed on this 
screen, as we see from the reverse shot (Figure 3.25). 

We see similar screens throughout the survey. Memorable examples from 
more recent sci-fi include The Fifth Element, Avatar, Mission to Mars, Minority 
Report, and Dollhouse (Figure 3.26). 

Part of the reason for the popularity of translucent displays in contemporary 
sci-fi may be that it allows directors and cinematographers to shoot actors 
through the screens, creating an interesting layered look that shows the 
actor's face and what they're looking at, simultaneously. 

These are distinguished from heads-up displays (see Chapter 8) in that the 
content on translucent displays is independent of the background behind 
it, where heads-up displays are intended to augment content behind it. 
Transparent displays whose content doesn't do any augmentation might be 
useful to allow the user an ambient awareness of their surroundings, but 
introduces visual noise, discussed below. 

Interface Layers 

We also see examples of interfaces that use data transparency to layer 
multiple levels of information on the screen, often for the purpose of 
conveying complexity and sophistication. 

Visual Interfaces 


FIGURE 3.26a-e 
Minority Report (2002); The Fifth 
Element (1997); Mission to Mars 
(2000); Dollhouse (2009); 
Avatar (2009). 

- ^ ~>TS 7 . 75 Sqgffijpfafifr. 


In the film Eagle Eye (on the edge of sci-fi, but the most illustrative example), 
the interface seen for the ubiquitous "national security" ARIIA system 
has overlaid fields filled with a translucent, darkening color on which text 
and data are rendered more legible than if the field was clear. In this case, 
transparency is used to focus the user's attention on what is most critical 
while preserving some of the data that gives it context (Figure 3.27). It allows 
some information to have high contrast and other information to blend 
together, as appropriate. 

FIGURE 3.27a-c 
Eagle Eye (2008). 


Chapter 3 

The 2009 reboot of Star Trek featured interfaces that combined information- 
dense, transparent layers of information with lots of moving information 
displays, giving the impression of overwhelming sophistication and 
complexity (Figure 3.28). 

In District 9, when Wikus steps into the alien battle suit, he is presented with 
a multilayered 3D interface with translucent, multicolored icons and data in 
the alien language (Figure 3.29). It's a rich interface, involving both gestural 
and heads-up display. (See Chapters 5 and 8 for more examples of these.) 
The information it presents is disorienting to him, partly because he doesn't 
understand the language, but also because the visual field is so dizzyingly busy. 

FIGURE 3.28 
Star Trek (2009). 

FIGURE 3.29 
District 9 (2009). 

Visual Interfaces 


Transparent displays and interface layers, indeed, offer expert users large 
amounts of information with which to make decisions, which can be a useful 
thing. But the amount of clutter might just as easily make the data that 
much harder to perceive or interpret. When transparency is combined with 
glowing data, the effect can be beautiful but even more disorienting. 


Transparency can blend layers of information together to 
show relationships or alignment. High opacity or fully opaque 
backgrounds should be used to draw users' attention to critical 
information whereas high transparency can be used to show 
more general, less important connections between sets of 


As described above, transparent layers can help organize and 
prioritize information, but too many become confusing and 
distracting. There's no optimal number of layers or level of 
transparency, but to make key information distinguishable, one 
guideline is that it should have a strong value contrast from 
its background. The Americans with Disability Act 5 provides a 
guideline of 70 percent difference for "detectable warnings," 
though if you are not concerned with universal accessibility you 
might comfortably halve that to 35 percent or higher depending 
on how fast the user is expected to perceive the information. 


It's common to see elements within interfaces— particularly windows and 
fields in gray— with characteristics that make them look more like physical, 
dimensional objects. These characteristics have been ubiquitous in real- 
world computer software since the 1990s, and they appear often in sci-fi as 
well: beveled edges on windows, frames, and buttons; gradations across the 
"flat" surface of objects to give them a sense of being illuminated by lights in 
3D space; reverse effects to make buttons and tabs look "debossed" into the 
surface when pushed; rulers scored with shaded lines across their surface, 
and so on. In some cases, text is rendered with light and dark edges to make 
it look etched into a surface (Figure 3.30). 

5 US Department of Justice. (1991). 1991 ADA Standards for Accessible Design. Retrieved from 

54 Chapter 3 

FIGURE 3.30 

The Matrix Revolutions (2003). 

Many of these effects are accessible for audiences because most people have 
seen and used them before in real-world interfaces, and additionally because 
these things work with people's sense of objects in the real world: real-world 
buttons can be pushed, and since this looks like a button, pressing it should 
activate it. There is some debate in the design community about the place 
of these skeuomorphs, but one thing is certain: these real-world references 
provide a fast cue to what can be done in an interface. 


A beveled appearance, drop shadow, or other simulation of 
physical attributes can make the interaction for on-screen con- 
trols more apparent and easier to understand as they appear 
like the physical controls they mimic. If the on-screen control 
doesn't function similarly to the physical control, however, 
greater confusion will result from the mismatch. 

Grouped Controls 

We see considerable variation in the groupings of controls like menus and 
toolbars. Current GUIs typically have commands grouped into menus at the 
top of appropriate application windows, but sci-fi offers some alternatives 
that remind us that this is merely a popular choice and not the only way to 
handle commands. 

The frame graphics in the LCARS interfaces serve to visually group controls 
of supposedly similar or related functions. This helps the crew parse the 
complexity of screens to find needed controls quickly and reliably. (For more 
detail about the LCARS, see the case study on page 68.) 

Visual Interfaces 


FIGURE 3.31a,b 

Alien (1979); Lifted (2006). 

Visual grouping needn't be limited to GUIs, however. Although several sci-fi 
properties intentionally make displays and controls indistinguishable from 
one another with rows and banks of nondescript lights and switches— as in 
the original Alien or the animated short, Lifted— some sci-fi properties use 
layout, color, and grouping to make the mechanical controls of interfaces 
more clear and understandable (Figure 3.31). 

The TV series Space: 1999 used this grouping principle for physical controls 
well, even though there are no touch screens or large display panels. 
Throughout Moonbase Alpha, all the controls are mechanical, but the panels 
they are attached to use considerable white space and colored backgrounds 
to group them together. Dark backgrounds are common, and color is 
used throughout, but in a way that suggests order and color coding. We 
don't know what all of the controls are meant to do but the colored bands 
and groupings do tell us which go together (Figure 3.32). The Aesthedes 
computer (discussed in Chapter 2) similarly groups controls into related 
functions that make the interface much easier to understand and use (see 
Figure 2.9). Though an ultimately unsuccessful computing platform, it is a 
good example of this grouping technique used well. 

J ■ \ I 



FIGURE 3.32a,b 
Space: 1999 (1975). 


Chapter 3 

Very few sci-fi interfaces are created by professional interface designers. 
Some are actually real products rarely seen by most audiences, as in the case 
of the spatial file system in Jurassic Park. Others come out of research labs, 
like the gesture-based interface and language in Minority Report, which is 
based on work from the MIT Media Lab. Only recently have professional 
interaction designers been asked to consult on sci-fi interfaces. 

One example is the film The Cell, for which interaction designer Katherine 
Jones was hired to create the interfaces. The screens are the control system 
for a brain/consciousness interface system, allowing two or three people to 
enter the consciousness of each other. These screens are carefully considered 
with few extraneous visuals elements and clear cues for interactivity. They 
stand out as an example of great on-screen grouping (Figure 3.33). 



One of the purposes of looking to sci-fi for inspiration and les- 
sons is that it often shows us examples of alternative approach- 
es to interfaces that we don't usually see. This doesn't mean 
that these different approaches will always be successful, but 
it's a duty of design to explore new arrangements of controls, 
displays, and interactions in order to evolve best practices and 
to find novel solutions that work better for users. 

WIMP (windows, icons, menus, and pointers) conventions are 
so ubiquitous that it's often easy to forget that this is just one 
way to deal with the large number of functions users might 
need to access at any time. With alternate inputs like voice 
command and gestural recognition as well as new outputs like 
3D and augmented reality, designers should be careful not to 
simply transfer metaphors from early GUI days, but to question 
them fundamentally and redesign where warranted. 

Visual Interfaces 


FIGURE 3.34a,b 
Iron Man (2008). 

File Management Systems 

When a sci-fi character needs to find a specific file or piece of data, or 
duplicate a file, they must access a computer file management system. 

As with many interfaces in sci-fi, much of what is visible in these interfaces 
is gibberish— words and symbols designed to fill the screen and meant 
to zoom by too quickly to be read— making these screens problematic to 
analyze with real-world principles. Additionally, many movies and TV 
shows wish to avoid showing actual file systems like Mac OS or Microsoft 
Windows, though those same systems need to be familiar enough so that 
no explanation is needed. Still, the few examples of these types of sci-fi file 
management systems provide some useful lessons. 

One example of such a system comes from Iron Man. When Tony Stark's 
assistant, Pepper Potts, is secretly searching for and copying files from 
Tony's computer at work, the files are represented with a slight 3D effect and 
translucently laid on top of each other. In this interface, the file currently 
being copied is larger in size than the rest of the group. This is the only 
indication of progress. At the same time, the contents of the file itself are 
shown on the screen (Figure 3.34). 


Chapter 3 


When presented with a lot of data or many options, only a few 
of which are meaningful, helping users quickly identify the im- 
portant ones is crucial. Varying the size of file representations 
such that the important ones are larger lets users rely on visual 
comparison, so they can pick them out quickly. 

A particularly lovely example of a novel file management system comes from 
the film The Final Cut. In it, the main character, Alan Hakman, is a video 
editor, hired by the family of a man who has recently died. His job is to create 
a tribute based on footage of his memories recorded directly from the man's 
implanted personal recording device. 

The editing machine he uses displays the vast amount of footage as visual 
stills representing snippets of life experience and arranges them along 
a timeline (Figure 3.35). There's more to the arrangement than merely a 
timeline, for the stills are stacked in sets, on top of each other, and grouped 
vertically. Transparency is used to create a sense of depth among the clips 
as well. Together, these effects create a stream of memories that Hakman 
traverses to find the ones relevant to the portrayal he creates. 

In a later scene, we see a different interface for his system that shows the 
edited clips he's selected to use in the final presentation. Here, the clips are 
grouped, labeled, and layered, but transparency is used only for the label, 
not the clips (Figure 3.36). Like many sci-fi films, this one doesn't explain the 
details of the interface, but the depiction offers us hints to visual techniques 
we can use to organize, relate, and display large amounts of data in a way 
that makes it easier to use. 

FIGURE 3.35 

The Final Cut (2004). 

Visual Interfaces 


FIGURE 3.36a-c 
The Final Cut (2004). 

In addition to the Jurassic Park example that opened the chapter, there 
are several, imaginative representations of 3D file systems in sci-fi. All of 
these seem more advanced than common 2D interfaces because they add 
a dimension for organizing files and data not present in 2D interfaces. Of 
course, whether these are truly more effective or not depends on many 
details of their design. 

In the film Gamer, the title character Simon has a room dedicated to his 
gaming and computer use with a volumetrically projected interface that 
surrounds him in a 360-degree, floor-to-ceiling virtual womb. Different 
files and content are accessed with different interfaces. In one, a 3D globe 
highlights geographical information of interest to him (Figure 3.37a). In 
another, a set of still images is clustered into a layered, 3D cloud (Figure 3.37b). 
In another, we see layers of messages to him from people all over the world 
(Figure 3.37c). We don't see all of these interfaces in use, but the layout is 
unique, interesting, and would take advantage of a user's full field of vision, 
body position memory, and spatial memory. These various interfaces surround 
him in simulated three-dimensional space even though most of the interfaces 
shown are two-dimensional. 

In the film Cowboy Bebop, scale is used to simulate depth within the 3D 
system. In the movies Hackers and Johnny Mnemonic, one-point perspective 
is used to simulate this axis in an almost architectural way. In these last two 
examples, only text is used to portray code and/or file names, whereas in 
Cowboy Bebop, images and icons are used exclusively to portray the content 
of files, almost as screenshots of the file itself (Figure 3.38). 


Chapter 3 

FIGURE 3.37a-c 
Gamer (2009). 

FIGURE 3.38a-c 

Johnny Mnemonic (1995); Hackers 

(1995); Cowboy Bebop (2001). 

Visual Interfaces 



Because we all live in a 3D world, we're already comfortable 
navigating 3D space. Most current file systems only arrange 
data in, at most, two visual dimensions (and very often only 
one dimension within a 2D window, as in a list view), with 
parent-child containers like folders and files. Using the third 
dimension may help users navigate and arrange more data in 
a smaller area and find it again more effectively. With physical 
objects, it's common for people to remember where things are 
spatially, using the surroundings as reference. Likewise, this skill 
can translate to data objects within a 3D file system. 

However, this only works if the spatial arrangement is consis- 
tent and doesn't change every time it is used. In addition, the 
layout can be more confusing for users who aren't spatially 
adept. If a user is the sort to frequently lose their keys, it would 
be problematic. 

It's also important that file details, such as file names or key 
words, aren't obscured. As the Jurassic Park example at the be- 
ginning of this chapter shows, the extra dimension can introduce 
as many new problems as it might solve, making it tricky to do 
well. The most successful examples we see are those that use 
the dimensions to orient files and data according to attributes of 
some kind and not merely another dimension in which files seem 
randomly distributed, as in the Cowboy Bebop example above. 

Motion Graphics 

There have always been blinking lights and buttons in sci-fi, but with touch 
screens and flat-panel displays, animation of the interface itself is becoming 
more common. By "motion graphics," we mean the interface itself, rather than 
the content it enables. For example, a folder icon that animates open when a file 
is dragged on top of it is an interface element, whereas the video that displays 
in a window in YouTube is not. In sci-fi, the data shown on display screens in 
Star Trek: The Original Series (see Figure 3.7a) is content as opposed to interface, 
whereas the folders in the Iron Man example above (see Figure 3.34), blinking 
and marching as their contents are copied, are part of the interface. 

For example, in Star Trek's LCARS interface, the frames and buttons of the 
interface rarely change as the user interacts (Figure 3.39a). A screen might 
be redrawn, with a new interface, to change its function (like opening a new 
application), but it's rare for animation to be incorporated into the use of the 
system. Video, charts, and animated content are sunk mostly into a content 
area of the overall LCARS interface. In contrast, screens in the film reboot 
Star Trek are more complex and often use animation within the interface 
itself (Figure 3.39b). 

62 Chapter 3 


^ JrS^^^F^r ^^^^^ '^ta^^^^B 

^&ffS & ^*&&^&^mA 

* / ty/t 


FIGURE 3.39a,b 

Sfar 7/-e/r; Voyager, "Equinox, Part 1" (Season 5, Episode 26, 1999); Star Trek (2009). 

In the film Lost in Space, many of the ship's interface screens are in constant 
motion. Most often, this motion is tied to useful information on the screen. 
But even if the content or controls are static, the background displays a 
pattern that holds the user's attention and/or conveys system information 
(Figure 3.40a). For example, movement in the background of the cryogenic 
interface indicates that the system is on and functioning (Figure 3.40b). 
Similarly, the motion in the background of file transfer interfaces indicates 
that the transfer is still processing while the foreground motion graphics 
indicate how much of the transfer has taken place (Figure 3.40c). The 
animations make the systems seem more dynamic. Their movement creates 
a very different effect than, say, the near-motionless LCARS interface. 

Visual Interfaces 



The human vision system has a separate track for drawing our 
attention called the superior colliculus. It is optimized to detect 
sudden light, sudden motion, and appearing/disappearing ob- 
jects very quickly with no conscious processing. Because of this 
system, moving graphics demand our attention very quickly. 

As in the Lost in Space examples above, movement in the back- 
ground of the interface can imply the system is active more 
than still imagery and graphics. But take great care, as this is an 
autonomic response that can distract the user from her present 
task. If she feels the forced switch of attention was not worth it, 
she'll become annoyed with the system, and rightly so. 


Motion graphics are more eye-catching and novel than static 
screens. Though they can add a sense of modernity, they can 
do more. They can add another layer of information: Whether 
the system is working or not, the current load on the network, 
or the system's confidence in the thing being developed, just 
to name a few examples. They can also imply the relation- 
ship of parts when transitioning between them: something is 
a superset or a subset of something else; things are similar or 
different to other things. Sci-fi rarely has the time to explain 
such nuances to its audience, but the real world has more time 
for the richness of an interface to unfold. For example, the 
transitions in programs like PowerPoint or Keynote are often 
gratuitous but, on occasion, are selected specifically because 
they describe a relationship between information that is clari- 
fied by the transition and not merely decorated by it. 

Visual Style 

Perhaps one of the most distinguishing aspects of sci-fi interfaces is their 
visual style — the combination of design elements like typography, color, 
shape, textures, layout and grouping, and even transparency that make a 
particular set of interfaces cohesive and unique. When done well, this visual 
style becomes a recognizable element across a film or series — almost as 
recognizable as some of the characters. 

Some of the makers of the earliest sci-fi films and TV shows, such as 
Metropolis and Flash Gordon, consciously created a visual manifestation 
of "the future" using specific visual elements. They did this with industrial 

64 Chapter 3 

as well as graphic design, and the results were critical to how audiences 
imagined and felt about the future, whether Utopian or dystopian. The 
interfaces in these shows reinforce the style. A crowded, dirty, or frenetic 
interface speaks to a world set against the protagonists. Similarly, a calm, 
easy-to-use, or elegant interface tells audiences and users that, in the future, 
technology will make their lives easier. 

For this section, we've chosen to focus on examples of movies or TV shows 
with strong visual styles that are not just well made but also help tell the 
story of the sci-fi worlds in which they appear. 

The Hitchhiker's Guide to the Galaxy 

The interface of the fictional electronic guidebook The Hitchhiker's Guide to the 
Galaxy, from the film of the same name, has a bold and humorous style that 
matches the feel of the film. Its backgrounds are full-screen, highly saturated 
color with a rainbow-colored interface system in which chosen topics slide out 
to the right from their peer groupings. Color is used to differentiate categories 
but doesn't seem to carry consistent meaning otherwise. This interface's many 
diagrams and animations are all in keeping with this flat, graphic style, with 
little hue variation within the subject category. It speaks of a world that is 
bold, friendly, a bit absurd, and unpanicked (Figure 3.41). 

FIGURE 3.41a-c 

The Hitchhiker's Guide to the Galaxy (2005). 

Visual Interfaces 


FIGURE 3.42a-c 
Final Fantasy (2001). 

Final Fantasy 

The adherence to a reference hue is used to a completely different effect 
in the animated film Final Fantasy. Here, only value is used to render the 
interface for analyzing a flowering plant, with large, subtle background 
images, overlaid text, boxes, and floating "windows." Transparency and 
gradation, as well as fine detail, create a completely unique feel of haunted 
technology (Figure 3.42). 

The Chronicles of Riddick 

In the cockpit of one of this film's ships we see a variety of physical controls 
and visual displays, including three central screens that project color 
graphics onto round plates of glass. The visual style uses flat and gradient 
areas of color (mostly blues and gray) with typography playing the primary 
role. Circles are the dominant graphic element in the interface, with an 
informational overlay of text and numbers, creating a technologically 
Spartan feel (Figure 3.43). 


Chapter 3 

FIGURE 3.43a-c 

The Chronicles of Rid dick (2004). 

The Incredibles 

In the animated film The Incredibles, the graphics and text are simple, clear, 
and sparse. Little filler is used, especially when Mr. Incredible accesses the 
villain's database to learn more about his evil plans. The use of flat graphic 
silhouettes and the Eurostile typeface are similar in style to Star Trek's (see 
the LCARS case study on page 68), but the graphic elements are set against a 
light blue and very light gray background, giving it the feel of a blueprint. As 
in several of the interface examples above, the colors are limited to a specific 
primary hue, except for the occasional, attention-grabbing, highly saturated 
red. This combination of elements gives the interface an appearance that is 
credible, distinctive, and original, reinforcing this alternate universe where 
superheroes became real in 1950s America (Figure 3.44). 

In the preceding examples, many of the elements used in the interfaces are not 
original. For example, Eurostile is a common sci-fi typeface, as are circular 
screens. Highly saturated palettes and wireframe models similarly appear in 
many films. It is the particular combination of these, used consistently, that 
gives these interfaces a unique style and makes them recognizable. 

A chapter on visual style would be incomplete if it didn't address the Star 
Trek LCARS interface. 

Visual Interfaces 



^^^V ^^^^^_ " i 







n» OMMt-OinccrtOr^At. TfiAfsie 

■- ' 17, ■ • . . 



^^ESHJ^ - 


FIGURE 3.44a-c 

7/7e Incredibles (2004). 

Case Study: Star Trek's LCARS 

One of the historically more significant shifts in sci-fi visual design occurred 
when Star Trek replaced controls based on mechanical, lighted buttons in the 
original TV series with the backlit touch panels in its first sequel series, Star 
Trek: The Next Generation. The new interface was striking not just for its visual 
distinctiveness but also for its comprehensiveness, extensibility, and influence. 

As mentioned in Chapter 2, the reason for this change was the TV series' 
budget: the creators simply didn't have the funds to recreate displays and 
controls made from so many separate buttons across the vast surfaces of 
the Enterprise's bridge. Production designer Michael Okuda and his team 
needed a much less expensive alternative. They looked to techniques such as 
those used in Logans Run nine years earlier, which featured sheets of plastic 
printed with graphics and backlit (Figure 3.45). 

Once Okuda's team decided on this technique, it freed them from the cost 
and constraints of physical interfaces, opening up tremendous design 
opportunities. Whereas this technique was used only for displays in Logans 
Run, however, in Star Trek: The Next Generation it was used for controls as 
well, presaging touch-screen technology. 


Chapter 3 

FIGURE 3.45 
Logan's Run (1976). 

The result is the computer interface called LCARS (Library Computer Access 
and Retrieval System). It consists of a black background with a condensed 
sans serif typeface (SWISS 911 Ultra Compressed BT) used throughout. It 
features rounded-corner background graphics in flat, pastel blues, purples, 
and oranges with areas of brighter, higher intensity color for both graphics 
and text. These swooping background graphics form frames along a grid, 
providing a structure into which buttons, labels, informational diagrams, 
and video are placed. This graphic system supported a wide variety of 
applications, diagrams, controls, and Starfleet technologies (Figure 3.46). 

The LCARS interface was so durable that it was used consistently across 
three Star Trek TV series and four films. It even inspired two thematic 
transformations, to its "past" and its "future," helping to reinforce the 
evolution of technology in this universe. 

To differentiate the interface of the prequel, Enterprise, a different visual 
language was created for the touch-screen displays mixed with physical 
controls (Figure 3.47). These displays were less integrated into panels and 
surfaces than those seen in the LCARS interface. The Enterprise interface may 
have been inspired by the LCARS interface (and foreshadow it), but it isn't 
technically part of it. The virtual portion of this interface used low-saturation 
color frames but buttons with more saturated colors, as well as a 2V2D shading 
effect on the on-screen controls that mimicked physical buttons. 

The Enterprise graphical language is almost entirely rectangular, and all 
data and controls are contained within closed, rectangular windows that 
do not overlap. There are very few round elements (though several of the 
window frames have a round "anchor" in the upper left corner), and a few 
rectangular buttons overlap their frames. These elements gave this series' 
interfaces a distinct yet related appearance while introducing limited touch- 
screen capability. 

Visual Interfaces 


FIGURE 3.46a-c 
LCARs interface from 
Star Trek: The Next 
Generation (1987). 

(MmEEmnMiuTi^^^^d ^^^^^^^^^^^^^mwmh* 




Si SS 

FIGURE 3.47a-c 
Enterprise 001 
interface from 


Chapter 3 

FIGURE 3.48a-c 
TCARS interface from Star Trek: 
Voyager, "Relativity" (Season 5, 
Episode 24, 1999). 

A future variant of the LCARS interface comes in the fourth TV series, Star 
Trek: Voyager. In the "Relativity" episode, a Starfleet ship from the future uses 
a modified LCARS interface called TCARS (Temporal Computer Access and 
Retrieval System; Figure 3.48). The TCARS interface has notable differences, 
although its lineage is clear: the black background panels persist, as do the 
touch screens. But the framing graphics use curved shapes based on long ovals 
and a predominantly blue hue. Several buttons have curved sides, based on 
ovals and not circles. In some screens, round elements establish radial grids 
instead of the perpendicular grids seen in the LCARS interfaces. In addition 
to the flat frame graphics, several are given dimensionality with shading, 
although with a softer effect than in Enterprise. 

These changes are enough to indicate the almost 200-year evolution of the 
TCARS interface from the fundamental elements of the LCARS interface. 
It serves to help audiences understand this leap in time without being 
confronted with something too unfamiliar. 

Visual Interfaces 


In addition to the LCARS interface found in Starfleet ships in Star Trek, 
many of the same underlying elements are used in the interfaces of other 
races seen throughout the 24th century, across the series (Star Trek: The Next 
Generation, Deep Space Nine, and Voyager) as well as the films associated 
with them. The large touch-screen surfaces with black backgrounds persist, 
but the colors, frame shapes, layouts, and typefaces are usually changed to 
make them look somewhat distinct. The overall effect is to make them all 
feel current with each other, of similar technology, but specialized to the 
cultures that use them (Figure 3.49). 

FIGURE 3.49a-c 

Kremen interface from Star Trek: Voyager (1997); Bajoran interface from 
Star Trek: Deep Space Nine (1998); and Borg interface from Star Trek: 
Voyager (1999). 


Chapter 3 



One of the most powerful ways to differentiate an interface, 
whether for sci-fi or real life, is to create an original color 
scheme, typographic treatment, and arrangement of displays 
and controls. This is no surprise to graphic or visual designers, 
but many clients (and directors) shy away from being original 
because they are leery of presenting their users or audiences 
with something too unfamiliar. However, as long as the visual 
elements are appropriate, don't create confusion, and facilitate 
understanding, this is one of the best ways to create a unique 
or differentiated product brand or interface experience. 

Visual Interfaces Paint Our Most 
Detailed Pictures of the Future 

Because television and film are such visual media, there is a great 
opportunity to study particular visions of the technology of the future. 
Even slight modifications to elements such as color, shape, symbolism, and 
typography can create drastically different screens than those we're used to 
seeing in real life today. 

These examples and the lessons throughout this chapter give us a 
vocabulary for remaking our real-world interfaces in order to make them 
look more futuristic. This could be useful in differentiating them from 
others, or to purposely relate them to sci-fi and the future. 

In addition, these lessons help us see that the boundaries of visual interface 
design are wider than we might have realized. Even if we don't intend for our 
interfaces to look like something from sci-fi, exploring these boundaries can 
help us discover and develop visual styles that are different than standard 
mobile, application, or computer interfaces, yet still look "real." 

Visual Interfaces 73 



What Counts? 76 

What Do Volumetric Projections Look Like? 78 

How Are Volumetric Projections Used? 81 

Real-World Problems 87 

Volumetric Projection Has Been Defined by Sci-Fi 90 


Forbidden Planet (1956). 

The swirling, luminescent smoke slowly settles into a form within the 
Krell transparent display case, taking the shape of a small version 
of Dr. Morbius's daughter. The figure smiles and shifts position. 
"That's Altaira!" Adams exclaims. Morbius explains calmly, "Simply a three- 
dimensional image, Commander." Despite this explanation, Adams looks on 
in amazement at the figure standing before him (Figure 4.1). 

This earliest example of a volumetric projection (VP) found in our survey 
was a modification of a very old illusion called Pepper's ghost, in which a 
bright, out-of-sight image is reflected off of a clean pane of glass in relative 
darkness, making the reflected image appear as if it is floating in space. It 
was thrilling for audiences and started the long love affair sci-fi makers have 
had with the use of VP. 

What makes these displays so appealing to sci-fi makers is that they are so 
cinemagenic. They are shape, light, and motion in a medium that works best 
with shape, light, and motion. Because they are not restricted to 2D screens, 
VPs can be put anywhere in a scene the sci-fi maker needs them to be. And 
because they're not real-world technologies, they are a quick, established 
way to signal a highly futuristic technology in a sci-fi story in a manner that 
is also useful to forward the plot. 

What Counts? 

"Volumetric projection" is a mouthful. Why not just call it what everyone 
else calls it — a hologram? Part of the reason is that that term already belongs 
to another kind of image. Remember those slightly 3D, multicolored shapes 
etched onto a silver substrate, as seen in the weird ending of Logan s Run 
(Figure 4.2a)? Or just look at a credit card — more likely than not there's a 
small hologram on it. This sophisticated printing technology has already 
laid claim to the term hologram, and these kinds of images are not like what 
we see projected in films. 


Chapter 4 

FIGURE 4.2a,b 

What doesn't count as a VP: Logan's Run (1976); The Last Starfighter (1984). 

Another potential name might be 3D display, but that term, too, has its 
problems. The ability to render objects on a 2D screen as if the objects were 
three-dimensional could be called 3D. It certainly was when computer- 
generated graphics first came on the scene, as in The Last Starfighter 
(Figure 4.2b). The term is further complicated by stereoscopic technologies 
that help a viewer perceive a display as if it were 3D, which are also sometimes 
included in descriptions of "3D displays," but that's not the same as VP either. 

The most specific term to describe what this chapter is about is volumetric 
projection — those massless, moving 3D images that are projected into space, 
which anyone can see with their own eyes from any direction without the 
aid of special viewing devices, such as glasses. This is really a long way of 
saying, "You know, like the Princess Leia 'Help me, Obi-Wan Kenobi' message 
in Star Wars" (Figure 4.3). That's certainly the most famous VP in sci-fi, and 
the most canonical. 

And although Star Wars is responsible for establishing the use (and the 
general appearance; see next section) of VP as the de facto communication 
medium in sci-fi, it's by far not the only example of it. VP is everywhere in 
sci-fi. It is such a staple of the genre that we don't have room to mention 
every movie or TV show that features it, much less to give an example of 
every type ofVP display. 


Star Wars Episode IV: A New Hope (1977). 

Volumetric Projection 


As in most other chapters in this book, some technologies skirt the 
boundaries of the technology discussed. Explaining why we include some, 
but not others, helps to identify the parameters of the topic. You may be 
wondering whether the Star Trek holodeck counts as VP. Savvy readers may 
have caught the inclusion of the term massless in our definition. But because 
nearly all of the holodeck's projections can have a perfectly detailed force 
field that gives them a sense of mass and solidity, this pushes them into a 
unique category. Therefore, the holodeck isn't considered in this chapter. 
(But you will find it discussed as a case study in Chapter 11.) 

What Do Volumetric 
Projections Look Like? 

When we look at the bulk of the examples, the first thing that strikes us is 
how similar they are. They tend to be translucent monochrome with a hint 
of blue, the whites glow a bit, and there are scan lines and the occasional 
flicker. The examples in Figure 4.4 show how strongly this style has remained 
in place since Forbidden Planet. 

Some examples take pains to add rays of light to the display to emphasize its 
dimensionality and point back to its source (Figure 4.5). 

■ ^ 

FIGURE 4.4a-g 

Total Recall (1990); Lost In Space (1998); The Matrix Reloaded (2003); 

Serenity (2005); Chrysalis (2007); Iron Man 2 (2010); Tron: Legacy (2010). 


Chapter 4 

FIGURE 4.5a-c 

Minority Report (2002); Star Wars 
Episode III: Revenge of the Sith 
(2005); District 9 (2009). 

1 ** 

fcl : t J 

\ Wyrw&. 





L ' "%_£■ 

FIGURE 4.6a-c 

Starship Troopers (1997); Star Wars 
Episode II: Attack of the Clones 
(2002); Avatar (2009). 

In some cases, the colors are supersaturated. This is most often the case 
when the VP is a wireframe model and the context is one of information 
display (Figure 4.6). 

In the rare cases in which the projection is not translucent, it's often 
because a trick is being played either on a character in the story or on the 
audience. One example is from Back to the Future Part II, when Marty McFly 
is surprised by a giant VP shark that "eats" him to advertise the fictional 
holofilm/awAS 19 (Figure 4.7a). Another is from the TV series Firefly when a 
brawler goes flying through a barroom "window" that only suffers a brief 
flicker instead of shattering (Figure 4.7b). This break in audience expectation 
says, "Ha! You thought it was real, but you forgot this is sci-fi!" 

Volumetric Projection 






FIGURE 4.7a, b 

Back to the Future Part II (1989); Firefly, "The Train Job" (Episode 2, 2002). 

This trick helps underscore why VPs need these visual traits in the first 
place. When you show something on screen that looks and moves like 
the real thing, how is an audience to know it's just a projection and not 
the real thing? How do you know, in a sci-fi world where the rules are by 
definition not the same as the real world, that this isn't some tiny clone of 
Leia who has just teleported in to speak to Luke? Of course, some of this is 
handled in context with dialogue, but the narrative overhead is reduced if 
the technology can be seen rather than explained. So, multiple cues help 
establish the virtualness of the VP beyond question. 


If a VP is informational and too realistic, the image, sound, or 
data in it may need to be altered so that users don't confuse it 
with reality. If for some reason the VP can't be altered, it should 
reveal its virtualness with a simple interaction, such as a ques- 
tion or the wave of a hand. 


If a VP is meant to deceive an audience, verisimilitude is impor- 
tant to its effectiveness. When through a simple interaction the 
user encounters the ruse, don't reveal the trick instantly. Wait- 
ing a beat builds up the joke so its reveal has more impact. 

Most of these cues adopt the visual artifacts of extant media. The peaked 
whites and bluish monochrome references black-and-white television. The 
scan lines and flicker reference broadcast television signal. More recent 
VPs adopt the edge-lit appearance of the electron micrograph. The use of 
projection rays references the projection light in a cinema. By adopting these 
visual conventions, the makers build on the audience's existing associations, 
helping them understand even more quickly and thoroughly that an object is 
only a projected image. 


Chapter 4 


The visual language of VP is established and narratively neces- 
sary. When this technology becomes commonplace in the real 
world, designers will have to break its dictates very carefully, 
or run into problems with understanding and acceptance by an 
audience trained across decades of sci-fi. 

How Are Volumetric Projections Used? 

The ways in which VPs are used vary more than their appearance. They 
are used for video messaging, navigation, tactical planning, advertising, 
entertainment, medical imaging, user interfaces, brain exercise, industrial 
design, and even alien abduction training, as seen in the 2006 Pixar short 
Lifted. The most common of these are communications, navigation, and 
medical imaging. 


Communications technology can be distinguished in several ways. One 
helpful distinction is whether the communication is synchronous or 
asynchronous. Synchronous technologies have the communicators interacting 
in real time, both attending the communication simultaneously, as on a 
telephone call. Most of the communications examples in the survey are 
synchronous (see Chapter 10). Asynchronous communications involve a sender 
encoding their communication in some medium, such as a letter, video, or 
audio recording, and then sending it to the receiver. The words asynchronous 
and synchronous are an eyeful, so we'll talk about the more friendly message 
and call, respectively. These two types overlap quite a bit, but where they don't 
is mostly in the composing, editing, and sending of messages. 

Most VPs in the survey are messages: the sender looks at the recording 
device, and the message is played back like a movie. But if it is a call, the 
speaker often looks directly at the recipient as if they were in the same 
space. This works for the TV or movie audience — after all, we'd be wondering 
what they were looking at if they weren't looking at the recipient — but on 
closer inspection, when the size or position of the VP is different than the 
sender would be in real life, it raises the significant problem of scaling and 
positioning. Let's take a look at this problem in detail, because although it 
appears to be about VP, its lessons can apply to any kind of video telephony. 

Take, for example, Darth Vader's projection in Star Wars Episode V: The 
Empire Strikes Back, when it appears in one of the AT- AT cockpits. The VP 
appears in miniature, probably to prevent a life-size image from blocking 
the stormtrooper pilot's view (Figure 4.8). 

Volumetric Projection 81 


Star Wars Episode V: The Empire Strikes Back (1980). 

Though this seems ideal 
from a "naturalness" 
perspective, it may be 
neither possible nor 
desirable. For instance, 
it would block the 
stormtrooper's view. 

Scaling may solve some 
problems from the 
stormtrooper's position . 

. . . but poses some strange 
problems from Vader's 
position. He wouldn't be 
physically or hierarchically 
comfortable with this. 


Neither simple cropping nor scaling solves the gaze-matching problem. 

For the stormtrooper, this miniaturization works well. He can glance down 
at a dashboard-component-size Vader when necessary. But what is Vader 
himself seeing? If the stormtrooper were to be scaled inversely — which 
would be necessary in order for Vader to make eye contact — Vader would 
need to be looking up at a figure scaled to gigantic proportions (Figure 4.9). 

To respect their relative status and make Vader physically comfortable 
using the communications system, he would need to be looking downward 
as well. Presuming the VP camera does not dynamically reposition itself to 
stay in the sight line of the speaker, the social need to look into each other's 


Chapter 4 

eyes introduces a gaze-matching problem. (This problem is called gaze 
monitoring in academic research.) As each looks at the eyes of the VP before 
him, he would appear to the other to be looking downward, as if avoiding eye 
contact. This is unacceptable, because it provides incorrect social cues of 
shyness, shame, subordination, or lying. 

Another possible way to solve the problem is to float the projections so that 
they are positioned at a level that matches normal gaze angles (Figure 4.10). 

Though this technique is used a few times in the Star Wars movies, in these 
cases care is taken not to show the free-floating VP sender's feet. In Revenge 
of the Sith, Obi-Wan is seen piloting a spaceship and conversing with a VP 
of Senator Organa. The filmmakers solved the floating person problem by 
cropping the movie frame so that the audience doesn't see Organa's feet 
(Figure 4.11). But in the real world, interface designers may not have this 
editing trick available to them, and in any case, floating projections may be 
detrimental in terms of secrecy or occlusion. 

Letting both participants glance downward 
would mismatch their gazes. 

Placing the scaled projection at a natural 
gaze position requires the VPs to float 
in midair, awkwardly and possibly 

FIGURE 4.10 

Scaling combined with floating partly solve the gaze-matching problem. 

FIGURE 4.11 

Star Wars Episode III: Revenge of the Sith (2005). 

Volumetric Projection 


So is scaling and repositioning in person-to-person VPs a nonstarter because 
of the gaze-matching problem? Not entirely. We just need to use apologetics to 
figure out how to make what works in the films work in real life. 


What if the VP technology is more complicated than it ap- 
pears? Whereas a camera passively conveys whatever infor- 
mation it picks up, the VP system is more like a scanner that 
blends photographic and computer-generated rendering for its 
output. It would work like this: the camera captures the send- 
er's movements, and then the system uses that information 
to construct a model of the sender that can be subtly altered. 
When it renders the sender's image for the receiver, it auto- 
matically adjusts body and eye positions so that eye contact 
can be made and gaze feels natural. When the real-world user 
matches the gaze of his VP co-communicator, his VP avatar on 
the other end of the call would match gaze with the real-world 
recipient, even if it requires adjusting the position and gaze of 
the avatar. 

It would have to be smart enough to recognize other social 
cues and let those pass through appropriately, such as lower- 
ing eyes in deference or rolling eyes in contempt. This way the 
position and scale of the VP can be adjusted, as the situation 
requires, while still maintaining natural social interactions. This 
is just as important in video conferencing, because the camera 
is rarely in the middle of the image we're given. Until we can 
situate the camera in the screen, at about eye level, we will 
need some kind of computational support to make the interac- 
tion look and feel more natural. 

This may be just what is happening when Sidious issues the infamous Order 
66 in Revenge of the Sith. A comparison of the images shows a mismatch 
in the speakers' positions: Commander Cody looks down into the eyes of 
a miniature VP Sidious, who is looking up at him, and the real Sidious is 
looking horizontally at the projection of Cody, who looks horizontally back 
at him (Figure 4.12). 

The creation of software sensitive enough to understand, alter, and represent 
gestural nuance without altering its meaning is a daunting task, to say the 
least. But as interface designers tackle similar problems in the real world, 
perhaps simpler steps can be taken first, such as simply repositioning the 
direction of the eyes. If such socially aware and capable systems cannot be 
made, there is another more direct way to solve the gaze-matching problem. 

84 Chapter 4 

FIGURE 4.12a,b 

Star Wars Episode III: Revenge of the Sith (2005). 

FIGURE 4.13 

Star Wars Episode III: Revenge of the Sith (2005). 


As mentioned above, the first rule of placement for Star Wars 
VPs seems to be "don't float people, and hide their feet from 
sight if you do." But given the constraints of space, this may 
make the avatars quite small and, in long conversations, strain 
the necks of communicators looking downward. To avoid this 
prolonged strain, adjust the height of the VP "stage" and the 
position of the user such that representations can meet one an- 
other's gaze comfortably, and speak "eye to eye" (Figure 4.13). 

Reinforcing Social Hierarchy 

Another aspect of the Star Wars VPs becomes apparent when we compare 
those used by the Empire and those used by the Jedi. The Empire's VPs 
are almost always shown scaled, with superiors scaled larger than their 
subordinates (Figure 4.14a). The Jedi council, in contrast, reinforces its 
egalitarian principles by making sure that, where possible, VPs in live 
deliberations are sized to appear as they would in real life (Figure 4.14b). 

Volumetric Projection 


FIGURE 4.14a,b 

Star Wars Episode V: The Empire Strikes Back (1980); Star Wars Episode III: 

Revenge of the Sith (2005). 



Once the problems of gaze matching are solved, scaling 
becomes a possible variable in communications displays. 
What scale is appropriate depends on the context. It could be 
something fixed, such as hierarchy or seniority relative to the 
meeting; for example, the White House press secretary may 
be larger than the reporters in a virtual briefing. Or it could be 
something dynamic, such as indicating popularity in a public 
contest or the current speaker in a business meeting. 


Another common use of VP is to show navigation of objects or people 
through space. These scenes hint at some of the promise of VPs — namely, 
that they can be seen from all sides, allowing representation of the full 
spatial context to increase the chances of solving spatial problems. 
Unfortunately, many of these scenes are written for characters to discuss 
the difficulties of reaching a destination and to telegraph plot points, rather 
than for problem solving or multiuser interaction (Figure 4.15). 

FIGURE 4.15a-c 

Lost in Space (1998); The Matrix 

Reloaded (2003); Avatar (2009). 


Chapter 4 

FIGURE 4.16a-c 

Lost in Space (1998); Chrysalis (2007); 

Firefly, "Ariel" (Episode 9, 2002). 

Medical Imaging 

The third most common use of VPs is for medical imaging, to provide 
noninvasive, real-time views of internal systems (Figure 4.16). Edge-lit, 
translucent, and color-coded rendering allows multiple organs in complex 
physical relation to be identifiable and observable. Medical imaging 
showcases the only collocated use of VP, with the projection and the 
thing being projected in the same space. These medical examples, like the 
communications ones described above, showcase a different promise of 
VP — that is, the re-representation of reality in a way that is more convenient 
for observation, study, and understanding. (See more about sci-fi medical 
interfaces in Chapter 12.) 

Real-World Problems 

VP is not going away in sci-fi. It's too established and cinemagenic. But there 
are problems with it in the real world that aren't addressed fully by sci-fi. We 
discuss the main issues below. 


As mentioned above, there is a risk of confusion when people see something 
that looks three-dimensional but is massless. They will naturally presume 
it's real. The Pepper's ghost visual style adopted by sci-fi certainly helps 
telegraph this difference, but if designers begin to stray from this style, it 
may cause confusion until some other takes its place. 

Volumetric Projection 



The Pepper's ghost style has its own problems, however. Translucency may 
be a quick signal of masslessness, but it also introduces issues of eyestrain. 
The additional light coming from behind the VP forces the user's eyes to 
work harder to distinguish between the light sources, especially if the user is 
looking at the projection for data. 


Objects exist in 3D contexts, but the surroundings may not be of interest 
to the user. For example, in the above scene from Firefly {see Figure 4.16b), 
River is reclining on a hospital bed, but that bed is rightly not included in 
the VP because the doctor is not concerned with it. In some cases, as in the 
Lost in Space and Avatar examples above (see Figures 4.15a, c), the context 
surrounding a particular planet or location may be critical to understanding 
the information. 

But at other times, just a hint of the context is useful. In Attack of the Clones, 
Yoda listens to Obi-Wan's VP report from Geonosis. In this projection, rain 
can be seen passing within Obi-Wan's silhouette, but not outside of it, even 
though we see the projection from different directions (Figure 4.17). Without 
this subtle context clue, Yoda might misconstrue Obi-Wan's stance as 
discomfort in delivering the message, or interpret his yelling as excitement 
rather than trying to hear himself above the noise of the rain. 

If systems can't interpret the context smartly, as they do in the Star Wars 
examples, they will need to crop objects in the scene. This could slice objects 
awkwardly or cause moving objects to appear and reappear. 

FIGURE 4.17 

Star Wars Episode II: Attack of the Clones (2002). 


Chapter 4 

FIGURE 4.18 
Lifted (2006). 


If a 3D crop includes the walls or ceiling of a room, or other large objects in 
the environment, these things may occlude the items of primary interest in 
a volumetric scene. Translucency or edge highlighting may help the user to 
ignore these extraneous objects, as seen in the Pixar short film Lifted, but 
this introduces issues of eyestrain as mentioned above (Figure 4.18). 


Because VP is still out of reach for most audiences to experience firsthand, 
sci-fi makers may lean on its cachet and inappropriately present information 
in three dimensions that would better be presented in two. These examples 
are, fortunately, few and far between, but they nonetheless may set 
inappropriate expectations that 3D is good for anything. 

In one such example from The Matrix Reloaded, workers in Zion Control 
monitor security and manage access through the city's gates. They do this in 
a virtual reality where information panels float in a confused, overlapping 
3D space around them. Though not technically VP, it is easy to imagine that 
most of this interface would be physically and visually easier to use in a 2D 
format, or at least one that appeared two-dimensional from the operator's 
perspective, as seen in Avatar (Figure 4.19). 

tm- . 

FIGURE 4.19a,b 

The Matrix Reloaded (2003); Avatar (2009). 

Volumetric Projection 


Volumetric Projection Has 
Been Defined by Sci-Fi 

VP has the main benefit of presenting information in a way that matches 
how humans sense most of the things in the world around them — in 
three dimensions. Our binocular vision, stereophonic hearing, and use 
of motion parallax are major inputs to understanding and interpreting 
the information that is contained in 3D space. VPs promise to bring this 
capability to bear in our digital interfaces. 

But until volumetric displays in the real world become cheap and 
ubiquitous, most of us will design for and experience it in sci-fi. Perhaps its 
continued presence there will help push it forward in ways that will make its 
eventual adoption in the real world smooth and usable. 

90 Chapter 4 



What Counts? 92 

The Canonical Gestural Interface: Minority Report 95 

Gesture Is a Concept That Is Still Maturing 97 

Hollywood's Pidgin 98 

Direct Manipulation 102 

Gestural Interfaces Have a Narrative Point of View 104 

Gestural Interfaces: An Emerging Language 108 


The Day the Earth 

Stood Still (1951). 

Returning to the interior of his mysterious spaceship, Klaatu waves his 
hand in front of a panel outside the ship's interior chamber and the 
door opens. He approaches a panel of transparent controls and waves 
his hand again. In response, the controls illuminate, disks begin to spin, and 
a circular screen pulses with a soft light. He begins a voice recording in his 
strange, alien tongue, "Imrae Klaatu naruwack. Macro puval baratu ludense 
empliccit . . ." (Figure 5.1). 

This scene from The Day the Earth Stood Still shows the first gestural controls 
in the survey. 

What Counts? 

Gestural controls allow users to provide input to a system with the free motion 
and position of their fingers, hands, and arms. Some systems require the 
gestures to be performed while in contact with a 2D surface, such as a touch 
screen, though most do not. Some require the user to wear gloves to help 
the system identify finger positions, though most do not. Nearly all involve 
volumetric projection, though a few, like the one Klaatu uses, do not. Because 
gestural interfaces are designed for "direct" manipulation, they almost always 
lack intermediary interface elements such as mice, pointers, and cursors. 

Given this, there are three technologies that don't quite count. 

The first is the exosuit, such as the loader horn Aliens or the armored 
personnel units from The Matrix Reloaded (Figure 5.2). Do they count as 
gestural interfaces? 

Certainly, the wearers of these exosuits gesture to move the arms and legs of 
the suit. But the interface for each is a set of mechanical controls, and this 
distinguishes them. They are, in effect, complex and highly ergonomic 

92 Chapter 5 

FIGURE 5.2a,b 

Aliens, detail (1986); The Matrix 

Reloaded (2003). 

levers, switches, and potentiometers, but not what the interaction design 
community currently calls a gestural interface. 

The second edge case is the holodeck from Star Trek. Its users gesture within 
its virtual worlds, interacting with the system while being completely 
unencumbered. Should it be considered? The objects and characters in 
its virtual worlds are for all practical purposes real to the user, and so the 
technology is no more or less "gestural" than the real world. To control the 
holodeck itself, crew members use a combination of spoken commands and 
a touch-screen wall panel called the arch (Figure 5.3). For these reasons, 
the holodeck cannot be classified as a purely gestural interface, so it isn't 
considered among the other examples here. 

FIGURE 5.3a-c 
Star Trek: The Next Generation, 
"Encounter at Farpoint" (Season 1, 
Episode 1,1987). 



FIGURE 5.4a-c 
A vatar (2009). 

The third case is the re-embodiment technology in Avatar. Although it is 
similar to the holodeck, and Jake Sully does gesture in his Na'vi body, these 
movements have no special meaning for controlling an interface. Since 
Jake's real body is lying motionless in a chamber while he's big and blue, it is 
more as if he has changed shape than used an interface (Figure 5.4). 

A fourth case bears consideration: touch-screen-based gestural interactions. 
These are gestures that the user performs on a 2D surface such as a video 
display (as on current smartphones and tablets). In Iron Man 2, Tony Stark 
uses touch gestures to do a quick background search on a job applicant 
and zoom in to a sexy photo he finds (Figure 5.5). Though such interfaces 
are more limited in the types of gestures that can be performed and there 
are a great deal fewer of them in the survey than the free-form variety, they 
certainly do count as gestural interfaces and are considered in this chapter. 

FIGURE 5.5a,b 
Iron Man 2 (2010). 


Chapter 5 

The Canonical Gestural 
Interface: Minority Report 

One of the most famous interfaces in sci-fi is gestural — the precog scrubber 
interface used by the Precrime police force in Minority Report (Figure 5.6). 
Using this interface, Detective John Anderton uses gestures to "scrub" 
through the video-like precognitive visions of psychic triplets. After 
observing a future crime, Anderton rushes to the scene to prevent it and 
arrest the would-be perpetrator. 

This interface is one of the most memorable things in a movie that is 
crowded with future technologies, and it is one of the most referenced 
interfaces in cinematic history. 1 

It's fair to say that, to the layperson, the Minority Report interface is 
synonymous with "gestural interface." The primary consultant to the 
filmmakers, John Underkoffler, had developed these ideas of gestural 
control and spatial interfaces through his company, Oblong, even before he 
consulted on the film. The real-world version is a general-purpose platform 
for multiuser collaboration. It's available commercially through his company 
at nearly the same state-of-the-art as portrayed in the film. 

Though this chapter references Minority Report a number of times, two 
lessons are worth mentioning up front. 

FIGURE 5.6a,b 
Minority Report (2002). 

1 In a quick and highly unscientific test, the authors typed [sci-fi movie title] + "interface" into 
Google for each of the movies in the survey and compared the number of results. "Minority 
Report interface" returned 459,000 hits on Google, more than six times as many as the runner- 
up, which was "Star Trek interface" at 68,800. 




Hollywood rumor has it that Tom Cruise, the actor playing John 
Anderton, needed continuous breaks while shooting the scenes 
with the interface because it was exhausting. Few people can 
hold their hands above the level of their heart and move them 
around for any extended period. But these rests don't appear 
in the film— a misleading omission for anyone who wants to use 
a similar interface for real tasks. Although a film is not trying 
to be exhaustively detailed or to accurately portray a technol- 
ogy for sale, demos of real technologies often suffer the same 
challenge. The usability of the interface, and in this example its 
gestural language, can be a misleading though highly effective 
tool to sell a solution, because it doesn't need to demonstrate 
every use exhaustively. 


The second lesson comes from a scene in which Agent Danny 
Witwer enters the scrubbing room where Anderton is working 
and introduces himself while extending his hand. Being polite, 
Anderton reaches out to shake Witwer's hand. The computer 
interprets Anderton's change of hand position as a command, 
and Anderton watches as his work slides off of the screen and 
is nearly lost. He then disregards the handshake to take control 
of the interface again and continue his work (Figure 5.7). 


H^*7fiJ*M3 ImM 

FIGURE 5.7a-d 
Minority Report (2002). 


Chapter 5 

One of the main problems with gestural interfaces is that the 
user's body is the control mechanism, but the user intends to 
control the interface only part of the time. At other times, the 
user might be reaching out to shake someone's hand, answer 
the phone, or scratch an itch. The system must accommodate 
different modes: when the user's gestures have meaning and 
when they don't. This could be as simple as an on/off toggle 
switch somewhere, but the user would still have to reach to flip 
it. Perhaps a pause command could be spoken, or a specific 
gesture reserved for such a command. Perhaps the system could 
watch the direction of the user's eyes and only regard the ges- 
tures made when he or she is looking at the screen. Whatever 
the solution, the signal would be best in some other "channel" so 
that this shift of intentional modality can happen smoothly and 
quickly without the risk of issuing an unintended command. 

Gesture Is a Concept That 
Is Still Maturing 

What about other gestural interfaces? What do we see when we look at them? 
There are a handful of other examples of gestural interfaces in the survey 
dating as far back as 1951, but the bulk of them appear after 1998 (Figure 5.8). 

FIGURE 5.8a-e 
Chrysalis (2007); Firefly, "Ariel" 
(Episode 9, 2002); Lost in Space 
(1998); The Matrix Reloaded (2003); 
Sleep Dealer (2008). 



Looking at this group, we see an input technology whose role is still 
maturing in sci-fi. A lot of variation is apparent, with only a few core 
similarities among them. Of course, these systems are used for a variety 
of purposes, including security, telesurgery, telecombat, hardware design, 
military intelligence operations, and even offshored manual labor. 

Most of the interfaces let their users interact with no additional hardware, but 
the Minority Report interface requires its users to don gloves with lights at the 
fingertips, as does the telesurgical interface in Chrysalis (see Figure 5.8a). We 
imagine that this was partially for visual appeal, but it certainly would make 
tracking the exact positions of the fingers easier for the computer. 

Hollywood's Pidgin 

Although none of the properties in the survey takes pains to explain exactly 
what each gesture in a complex chain of gestural commands means, we can 
look at the cause and effect of what is shown on screen and piece together 
a basic gestural vocabulary. Only seven gestures are common across 
properties in the survey. 

1. Wave to Activate 

The first gesture is waving to activate a technology, as if to wake it up or gain 
its attention. To activate his spaceship's interfaces, Klaatu passes a flat hand 
above their translucent controls. In another example, Johnny Mnemonic 
waves to turn on a faucet in a bathroom, years before it became common in 
the real world (Figure 5.9). 

FIGURE 5.9a-c 
Johnny Mnemonic (1995). 


Chapter 5 

2. Push to Move 

To move an object, you interact with it in much the same way as you 
would in the physical world: fingers manipulate; palms and arms push. 
Virtual objects tend to have the resistance and stiffness of their real-world 
counterparts for these actions. Virtual gravity and momentum may be 
"turned on" for the duration of these gestures, even when they're normally 
absent. Anderton does this in Minority Report as discussed above, and we see 
it again in Iron Man 2 as Tony moves a projection of his father's theme park 
design (Figure 5.10). 

FIGURE 5.10a,b 
/ro/? Ma/? 2 (2010). 

3. Turn to Rotate 

To turn objects, the user also interacts with the virtual thing as one would in 
the real world. Hands push opposite sides of an object in different directions 
around an axis and the object rotates. Dr. Simon Tarn uses this gesture to 
examine the volumetric scan of his sister's brain in an episode of Firefly 
(Figure 5.11). 

FIGURE 5.11a,b 

Firefly, "Ariel" (Episode 9, 2002). 



4. Swipe to Dismiss 

Dismissing objects involves swiping the hands away from the body, either 
forcefully or without looking in the direction of the push. In Johnny Mnemonic, 
Takahashi dismisses the videophone on his desk with an angry backhanded 
swipe of his hand (Figure 5.12). In Iron Man 2, Tony Stark also dismisses 
uninteresting designs from his workspace with a forehanded swipe. 

FIGURE 5.12a-c 
Johnny Mnemonic (1995). 

5. Point or Touch to Select 

Users indicate options or objects with which they want to work by pointing 
a fingertip or touching them. District 9 shows the alien Christopher Johnson 
touching items in a volumetric display to select them (Figure 5.13a). In 
Chrysalis, Dr. Briigen must touch the organ to select it in her telesurgery 
interface (Figure 5.13b). 

FIGURE 5.13a,b 

District 9 (2009); Chrysalis (2007). 


Chapter 5 

6. Extend the Hand to Shoot 

Anyone who played cowboys and Indians as a child will recognize this 
gesture. To shoot with a gestural interface, one extends the fingers, hand, 
and/or arm toward the target. (Making the pow-pow sound is optional.) 
Examples of this gesture include Will's telecombat interface in Lost in Space 
(see Figure 5.8c), Syndrome's zero-point energy beam in The Incredibles 
(Figure 5.14a), and Tony Stark's repulsor beams in Iron Man (Figure 5.14b). 

FIGURE 5.14a,b 

The Incredibles (2004); Iron Man (2008). 

7. Pinch and Spread to Scale 

Given that there is no physical analogue to this action, its consistency across 
movies comes from the physical semantics: to make a thing bigger, indicate 
the opposite edges of a thing and drag the hands apart. Likewise, pinching 
the fingers together or bringing the hands together shrinks virtual objects. 
Tony Stark uses both of these gestures when examining models of molecules 
in Iron Man 2 (Figure 5.15). 

Though there are other gestures, the survey revealed no other strong 
patterns of similarity across properties. This will change if the technology 
continues to mature in the real world and in sci-fi. More examples of it may 
reveal a more robust language forming within sci-fi, or reflect conventions 
emerging in the real world. 

FIGURE 5.15a,b 
Iron Man 2 (2010). 





In the real world, users have some fundamental interface con- 
trols that movies never show, but for which there are natural 
gestures. An example is volume control. Cupping or cover- 
ing an ear with a hand is a natural gesture for lowering the 
volume, but because volume controls are rarely seen in sci-fi, 
the actual gesture for this control hasn't been strongly defined 
or modeled for audiences. The first gestural interfaces to ad- 
dress these controls will have an opportunity to round out the 
vocabulary for the real world. 


If these seven gestures are already established, it is because 
they make intuitive sense to different sci-fi makers and/or 
because they are beginning to repeat controls seen in other 
properties. In either case, the meaning of these gestures is 
beginning to solidify, and a designer who deviates from them 
should do so only with good reason or risk confusing the user. 

Direct Manipulation 

An important thing to note about these seven gestures is that most are 
transliterations of physical interactions. This brings us to a discussion 
of direct manipulation. When used to describe an interface, direct 
manipulation refers to a user interacting directly with the thing being 
controlled— that is, with no intermediary input devices or screen controls. 

For example, to scroll through a long document in an "indirect" interface, such 
as the Mac OS, a user might grasp a mouse and move a cursor on the screen to 
a scroll button. Then, when the cursor is correctly positioned, the user clicks 
and holds the mouse on the button to scroll the page. This long description 
seems silly only because it describes something that happens so fast and that 
computer users have performed for so long that they forget that they once had 
to learn each of these conventions in turn. But they are conventions, and each 
step in this complex chain is a little bit of extra work to do. 

But to scroll a long document in a direct interface such as the iPad, for 
example, users put their fingers on the "page" and push up or down. There 
is no mouse, no cursor, and no scroll button. In total, it takes less physical 
and cognitive work to scroll with the gesture. The main promise of these 
interfaces is that they are easier to learn and use. But because they require 
sophisticated and expensive technologies, they haven't been widely available 
until the past few years. 

102 Chapter 5 

In sci-fi, gestural interfaces and direct manipulation strategies are 
tightly coupled. That is, it's rare to see a gestural interface that isn't direct 
manipulation. Tony Stark wants to move the volumetric projection of his 
father's park, so he sticks his hands under it, lifts it, and walks it to its new 
position in his lab. In Firefly, when Dr. Tarn wants to turn the projection of 
his sister's brain, he grabs the "plane" that it's resting on and pushes one 
corner and pulls the other as if it were a real thing. Minority Report is a rare 
but understandable exception because the objects Anderton manipulates 
are video clips, and video is a more abstract medium. 

This coupling isn't a given. It's conceptually possible to run Microsoft 
Windows 7 entirely with gestures, and it is not a direct interface. But the 
fact that gestural interfaces erase the intermediaries on the physical side of 
things fits well with erasing the intermediaries on the virtual side of things, 
too. So gesture is often direct. But this coupling doesn't work for every need 
a user has. 

As we've seen above, direct manipulation does work for gestures that involve 
physical actions that correspond closely in the real world. But, moving, 
scaling, and rotating aren't the only things one might want to do with 
virtual objects. What about more abstract control? 

As we would expect, this is where gestural interfaces need additional 
support. Abstractions by definition don't have easy physical analogues, and 
so they require some other solution. As seen in the survey, one solution is to 
add a layer of graphical user interface (GUI), as we see when Anderton needs 
to scrub back and forth over a particular segment of video to understand 
what he's seeing, or when Tony Stark drags a part of the Iron Man exosuit 
design to a volumetric trash can (Figure 5.16). These elements are controlled 
gesturally, but they are not direct manipulation. 

FIGURE 5.16a-c 
Minority Report 
(2002); Iron Man 



'■ J , 



, Ahhi 



Invoking and selecting from among a large set of these GUI tools can become 
quite complicated and place a DOS-like burden on memory. Extrapolating 
this chain of needs might very well lead to a complete GUI to interact with 
any fully featured gestural interfaces, unlike the clean, sparse gestural 
interfaces sci-fi likes to present. 

The other solution seen in the survey for handling these abstractions is the 
use of another channel altogether: voice. In one scene from Iron Man 2, Tony 
says to the computer, "JARVIS, can you kindly vacuform a digital wireframe? 
I need a manipulable projection." Immediately JARVIS begins the scan. Such 
a command would be much more complex to issue gesturally. Language 
handles abstractions very well, and humans are pretty good at using language, 
so this makes language a strong choice. (See Chapter 6 for further discussion.) 

Other channels might also be employed: GUI, finger positions and 
combinations, expressions, breath, gaze and blink, and even brain interfaces 
that read intention and brainwave patterns. Any of these might conceptually 
work but may not take advantage of the one human medium especially 
evolved to handle abstraction— language. 



Gestural interfaces are engaging and quick for interacting in 
"physical" ways, but outside of a core set of manipulations, 
gestures are complicated, inefficient, and difficult to remember. 
For less concrete abstractions, designers should offer some 
alternative means, ideally linguistic input. 

Gestural Interfaces Have a 
Narrative Point of View 

Gestural interfaces can be distinguished by their narrative point of view. 
Interfaces such as the one Tony Stark uses to design the Iron Man exosuit 
and Chrysalis's telesurgical interface are second person, with the user 
manipulating states of objects or data. The survey shows several examples of 
gestural interfaces that are first person, in which users control a device, such 
as a robot, as if they were embodying the thing. 

In Lost in Space, Will uses a handheld device to control the family robot and 
remotely "joins" the adults investigating a potentially deadly spacecraft. When 
metallic, spidery aliens attack the group, Will realizes he needs faster control 
than the handheld can provide. He drops it and switches to a gestural control 
that is a first-person interface. It allows him to stand within a translucent, 
color-coded volumetric projection of the robot and control its direction, speed, 
gaze, arms, and weapons systems (see Figure 5.8c). 

104 Chapter 5 

— • 

I ' r - 

\ '* 



If***-* £k 

^ >- 

FIGURE 5.17a,b 
5/eep Dealer (2008). 

•■■ .__ 1. 

*** *CP 

L BL1 


1 \ 

FIGURE 5.18a,b 
S/eep Dealer (2008). 

In the movie S/eep Dealer, Memo lives in Tijuana, Mexico, but does 
construction work remotely in San Diego using a "wetware" interface 
(Figure 5.17a) to control a small robot (Figure 5.17b). His gestures control its 
movement, gaze, and arms, as well as a welding arc. 

In this same film, Rudy works for the US Air Force using a similar wetware 
interface (Figure 5.18a) to gesturally control drone attack planes that patrol 
Mexican territory looking for rebels (Figure 5.18b). 

An interesting and unusual example appears in Johnny Mnemonic, when in 
a video call the deceptive businessman Takahashi gesturally controls the 
computer-generated image of a person he's had killed. To do this, he holds 
his hand above a scanner and moves it as if it was the mouth of a puppet. 
The computer interprets this gesture and moves the lips of the avatar as a 
result. It also disguises his voice to sound as if he were the dead man. It is a 
unique example in that the avatar is mapped to a body part rather than to 
Takahashi's entire body (Figure 5.19). 

For the most part, these first-person interfaces might be considered more 
natural than the second-person gestural interfaces. The remotely controlled 
robots are an extension of the user's body, with clear mapping between the 
user and the avatar. 



FIGURE 5.19a-c 
Johnny Mnemonic (1995). 

Complications arise from the mismatch of the thing being controlled and 
the body doing the controlling. In some cases, the human can do something 
that the machine can't. Neither of the robots mentioned above could jump, for 
example, though there's nothing to stop their controllers from doing just that. 
This is a relatively simple problem, as the system can simply ignore the input. 

What's more troublesome is when the robot can do something for which 
the human doesn't have an easy analogue. In Lost in Space, we see the robot 
retreating from the spidery aliens. Will extends his hands in the direction 
of the spiders to fire the robot's weapons at them. Cutting back to a view of 
the robot, we also a see laser shooting from the robot's head. How does Will 
control that? 

It's simple to come up with an apologetic explanation— that the laser is 
simply following his gaze, or it is controlled by systems usually controlling 
the entire robot. But the mismatch is a good illustration of the core problem 
of first-person interfaces. What would Will do if he wanted to control the fire 
of that third laser? His hands and legs are otherwise engaged. Some other 
channel of input must be engaged to address this additional control. 

The problem becomes even more complex when the robot is less 
anthropomorphic. In Sleep Dealer, Rudy's gestural interface controls a 
drone aircraft. What's the gestural analogue of dropping bombs? Or an 
Immelmann turn (Figure 5.20)? These things don't have intuitive analogues. 

Rudy could use a gestural throttle, of course, but this just illustrates the limits 
and challenges of a full first-person gestural interface. It can't do everything 
that the robot can, and therefore needs additional interface layers. 


Chapter 5 

FIGURE 5.20 

An Immelmann turn. 



Third-person gestural controls allow control of the behavior 
of a thing from without. This could be as simple as a camera 
above and just behind an avatar. Many video games, and virtual 
worlds such as Second Life, adopt this convention to allow 
for a better view of the avatar's environment. A third-person 
perspective can also describe a bird's-eye view that would, for 
example, allow a general looking at a live, sand-table view of a 
battlefield to control an army of robots by means of gestures 
and commands. 

The written documentation for the Minority Report Precrime 
scrubber explains that to select a new camera angle, users 
indicate with their left hand the thing to be viewed, and use 
their right hand to describe the position and frustum of the 
camera. Though this is a third-person gesture, it is not apparent 
from watching the movie, and the survey didn't find any other 
examples of a third-person gestural interface. 


First-person gestural interfaces work best as an extension of 
the user's body when the controlled device is anthropomor- 
phic. Otherwise, an additional layer of interface is required and 
might call on another point of view altogether. Sometimes, 
giving the user the ability to swap between views may accom- 
modate different tasks. 



Gestural Interfaces: 
An Emerging Language 

Gestural interfaces have enjoyed a great deal of commercial success over 
the last several years with the popularity of gaming platforms such as 
Nintendo's Wii and Microsoft's Kinect, as well as with gestural touch devices 
like Apple's iPhone and iPad. The term natural user interface has even been 
bandied about as a way to try to describe these. But the examples from sci-fi 
have shown us that gesturing is "natural" for only a small subset of possible 
actions on the computer. More complex actions require additional layers of 
other types of interfaces. 

Gestural interfaces are highly cinemagenic, rich with action and graphical 
possibilities. Additionally, they fit the stories of remote interactions that are 
becoming more and more relevant in the real world as remote technologies 
proliferate. So, despite their limitations, we can expect sci-fi makers to 
continue to include gestural interfaces in their stories for some time, which 
will help to drive the adoption and evolution of these systems in the real world. 

108 Chapter 5 


Sonic Interfaces 

What Counts? 


Sound Effects 


Ambient Sound 


Directional Sound 


Music Interfaces 


Voice Interfaces 


Sonic Interfaces: Hearing Is Believing 



The Day the Earth 

Stood Still (1951). 

After the alien Klaatu is shot by a nervous soldier, his robot 
companion, Gort, appears menacingly from within the landed 
spacecraft. Its visor slowly rises to reveal a disintegration beam, 
which Gort uses to begin destroying all weapons in sight, including artillery 
and a tank. Klaatu, wanting to de-escalate the situation, turns to Gort and 
shouts, "Gort! Declet ovrosco!" In response, Gort stands down, ceasing the 
counterattack (Figure 6.1). 

Gort isn't the first piece of technology in sci-fi to have a conversational 
interface. That honor belongs to the wicked robot Maria from Metropolis, 
but as it is a silent film, we do not get to hear the commands delivered to 
her. In the case of The Day the Earth Stood Still, we can study the tone, pace, 
and responses in real time without having to interpret from lip reading 
and intertitles. This makes it easier to study this primary example of a 
sonic interface. 

What Counts? 

Sound becomes part of the interface when it is an input or output for a 
system's state and function. Note that this is distinguished from simple 
audio content, such as the music from a radio. We've broken out sonic 
interfaces into two broad categories: sonic output and voice interfaces. 

Sound Effects 

We commonly encounter systems that use sounds for output: status, alerts, 
and responses. For example, our telephones play a distinct tone for each 
button pressed in the numeric keypad. Alarm clocks buzz to wake us up. Cars 
chime to remind us to buckle our seat belts. We see similar system sounds 
throughout sci-fi, as well. Audiences have come to expect some kind of audio 
interface because it helps us understand the action in a film or TV show. 

110 Chapter 6 

A Brief Experiential History 

The ringing of a telephone was one of the first sonic interfaces common in 
people's lives, but even though the telephone appeared in the late 1800s, it 
remained one of the few sound interfaces until well into the 1950s. Though 
they produced sound to deliver content, even radio and television didn't 
deliberately employ sound in their interfaces until much later. Sound effects of 
this time were analog and mostly confined to appliances such as alarm clocks, 
buzzers on ovens, and bells on timers. 

From the production side of things, beeps and buzzes, chimes, and tones can 
be added to a soundtrack along with all of the other sounds necessary in a 
TV show or movie, such as the click of a button or the creak of a door. The art 
of adding such sounds is called Foley after Jack Foley, a sound engineer who 
launched the field in 1927, and is surprisingly more complex than one might 
initially think. For example, almost every sound other than actors' voices in a 
movie is added after the filming is complete, 1 and many sounds are created 
using objects that differ from those being portrayed. Despite this complexity, 
sound plays such a strong role in conveying a sense of realism and futuristic 
technology that studios have included simple effects in sci-fi as part of sonic 
interfaces since the advent of talking pictures. 

1 And sometimes even the actors' voices are recorded in a studio and put back into the 
soundtrack of the film. These must be precisely synchronized with the original speech 
and actors' lips. 

For example, when we hear the specific double beep with rising tone in Star 
Trek, we know the system is hailing someone. Similarly, when we hear the 
same tones reversed we know that the communication is over and the channel 
is now closed. If the same beep were used in each case, it would confuse us and 
the characters as to whether the channel was still opened or closed. 


Users need to be able to differentiate system sounds to un- 
derstand their meaning. Systems that use multiple sounds and 
sound sequences to communicate system messages will require 
some learning, but ultimately they communicate more informa- 
tion. In addition, the sounds need to be used consistently with 
specific actions in order to be associated with those actions. 

The later Star Trek episodes used nearly twice as many different 
systems sounds, sequences, and voice responses as the first series. This 
differentiation could speak to the sophistication of the ship's systems in 
specifying audio output with more precision, the care the production 

Sonic Interfaces 111 

designers took and the increased sophistication of the tools available to 
them, and the audience's increasingly sophisticated expectations and 
understanding of system sounds in the interface. Regardless, because many 
of our expectations are set or influenced by what we see in media, developers 
must consider more sophisticated sound solutions in their interfaces. 

Ambient Sound 

The ambient clacking of moving parts within a mechanical computer, like 
reels turning to access a section of tape memory, can be considered part of 
an interface because the clicks indicate that the system is working, even 
though this sound is mostly a by-product and not a designed signal. 

In sci-fi, we find numerous examples of computer systems making such 
sounds, particularly to signal to audiences that they're working to process 
a large set of data. When Scotty, the chief engineer of the Enterprise on Star 
Trek's original series, remarks that he can tell that the ship's engines aren't 
tuned correctly because the hum they are producing is slightly off, he is calling 
attention to such sounds. But this doesn't have to be an accidental by-product. 
With digital technologies, we can include this information deliberately. 


Different ambient sounds can unobtrusively inform a user that 
a system is operating and indicate its current state in broad 
strokes. Ambient sounds need to strike a balance between be- 
ing the sonic focus and being too far in the background. If the 
sounds are completely unobtrusive, they aren't useful. To be ef- 
fective, they must not come to attention until it's required, like 
when a system problem arises. This means the level of sounds 
must be calibrated, beforehand or dynamically, so that back- 
ground sounds can come forward to a user's attention. 

Directional Sound 

Humans naturally hear in three dimensions. Our ears are extremely sensitive, 
capable of discerning microsecond differences between the sound waves 
reaching each of our ears. Systems that produce sound directionally can 
enhance our understanding of where a sound source is in space, its direction, 
and its speed. Because our sense of directionality is fast and subconscious, it 
must be done precisely when replicated technologically, but the effects provide 
information to users that are immediately understandable and actionable. An 
apologetics example helps explain its power. 

112 Chapter 6 


Star Wars Episode IV: 

A New Hope (1977). 

When Luke and Han climb into their gunner stations aboard the Millennium 
Falcon, they strap themselves in, turn on the targeting computer, and put 
on headphones. As TIE fighters speed by, we hear the roaring approach of 
their engines, the piercing blasts of the laser cannon fire, the fading zoom as 
they speed away, and, if the stormtroopers are less lucky, the boom as their 
ship explodes (Figure 6.2). Few people pause to consider the physics of the 
situation, but where are these sounds coming from? After all, there is no air 
in space to convey sound waves between the exploding TIE fighter and the 
Falcon. Of course we could excuse this as a convention of film, a way that the 
filmmakers engage the audience in the firelight. But if it helps the audience, 
wouldn't these same sounds help the gunners, too? What if this wasn't a 
filmmaker's trick but a powerful feature of the weapon system itself? Let's 
presume that the Falcons sensors are tracking each TIE fighter in space and 
producing the roars and zooms directionally to provide a layer of ambient 
data that helps the gunner track opponents, even when there are several 
targets or they are out of sight. This makes the sound effects a powerful sonic 
aspect of a mission-critical system. 



Hearing people don't have to learn to locate sounds in space 
directionally. It's a built-in capability. Designers can use this to 
place information, even when it's not "naturally" spatial, in the 
space around the user. For example, if an interface for moni- 
toring stock market portfolios used sounds to draw attention 
to trading activity that was likely to affect the portfolio, these 
signals could be made to seem closer than sounds used to 
indicate other activity. A user might need to be trained for 
the meanings behind arbitrarily assigned directions, but there- 
after they would provide contextual clues to help inform more 
concrete tasks. 

Sonic Interfaces 


Directional sound is a little tricky to portray to more than one user in a 
space and works best when the sound delivery is spatially constrained, 
such as with headphones or in a small area occupied by one person. When 
more than one person inhabits a space, their individual capabilities, 
orientations, and locations need to be processed so accurate sound can be 
sent to each one separately, which may be prohibitively complicated for most 
multiuser systems. 

Music Interfaces 

We see only two interfaces in the survey that use music as a part of the 
interface as well as part of the content. The first example is Close Encounters 
of the Third Kind, in which a specific tonal sequence forms a welcome 
message— one of acknowledgment and understanding. The five tones 
are G, A, F, F (an octave lower), and C. This musical phrase is implanted 
telepathically into a few people who encounter smaller alien spacecraft as 
an invitation to visit the massive mother ship that arrives at the climax of 
the film. When the mother ship appears at Devil's Tower, the US Army greets 
it with the same tones played on a specialized electronic organ (Figure 6.3). 

It is a simple musical interface, with a user playing a standard synthesizer 
keyboard. As each note sounds, a corresponding colored light illuminates on 
a huge array. This is the visual part of the alien language, which is vital for 
complete communication. 

The other example is an interface from Barbarella that uses music as a 
weapon. Like in Close Encounters of the Third Kind, the music is part content, 
part interface. Here, the evil scientist Durand-Durand straps Barbarella into 
a seat within his musical torture device called the Excessive Machine. Each 
note he plays on the keyboard simultaneously performs nefarious sexual 
acts on its victims, in an attempt to pleasure them to death. Though the 
exact cause and effect is demurely hidden from view, it is worth noting for 
the synergy of the playing, the music, and the intent (Figure 6.4). 

FIGURE 6.3a-c 

Close Encounters of the Third Kind 




— ^ 








Chapter 6 

Barbarella (1968). 



What if interfaces could use music for indicating system status? 
It would not be without its challenges: encoding meaning into 
the music, handling users' preferences for different styles of 
music, and processing data aesthetically so that its patterns 
are intelligent and not cacophonous. But once solved, it might 
be a way to receive system information— particularly ambient 
information— that is pleasant and not widely explored. 

Voice Interfaces 

Where sonic interfaces really strike a chord is with voice interfaces. They are 
found throughout sci-fi, and over time their frequency has been increasing, 
for a number of reasons: 

• Voice interfaces are easy for writers to imagine and audiences to 
understand because they rely on our innate language abilities. 

• Voice interfaces are easy to create and portray. For input, actors speak into 
a system and it responds accordingly. For output, a voice actor can provide 
the output, and the track incorporated as part of the Foley editing. 

• Voice interfaces still seem advanced, especially when they have fully 
conversational capabilities. 

Sonic Interfaces 


Voice interfaces have a range of sophistication: 

• Simple voice output provides spoken information for listeners. 

• Voice-identification systems, most often used for secure access, 
responds to specific words or numbers for security and can sometimes 
include voiceprint analysis. 

• Limited-command voice interfaces might only respond to a few words 
and phrases and a small set of commands, making it easy for systems 
to isolate and identify the correct meanings. These can be coupled with 
voice-identification systems, in some cases. 

• Conversational interfaces are capable of parsing everything spoken 
by a speaker. This is a humanlike level of language understanding. 
Conversational voice interfaces include both language recognition and 
voice synthesis. 

In current, real-world interfaces, these four categories are fairly distinct. 
In sci-fi, however, we find examples with a combination of these systems. 
As technology progresses, this may point the way toward a greater mixing 
within real-world interfaces, as well. 

Simple Voice Output 

The most basic of voice interfaces are those that provide information spoken 
as a prerecorded or synthesized voice. 

One of the most familiar of the simple voice outputs in sci-fi is the warning 
countdown. In movies including Alien, Star Trek, and Galaxy Quest, when 
scuttling procedures are initiated, a Klaxon sounds as red lights provide a 
general warning. While attention getting, these mechanisms only tell that 
there is a critical problem, not exactly what that problem is or what is to 
be done about it. To provide that information, a female voice repeatedly 
announces that the self-destruct is under way, and that people have a 
certain amount of time to evacuate to a safe location. 


Should a bit of system information be conveyed through audio 
or spoken language? Though every piece of content needs to 
be considered in its particular context of use, a good rule of 
thumb is to put peripheral information in peripheral channels. 
Since Captain Kirk could feel confident that his communica- 
tor was on when he heard voices coming from it, and off when 
he heard nothing or got no response, the double beeps that 
signal opening and closing communications can be considered 
peripheral and can be signaled as system sounds instead of 

116 Chapter 6 

a voice that would convey the same information. Conversely, 
if the system needs to convey that 10 seconds remain before 
the whole place is gonna blow, it could be conveyed as a rising 
tone, but the information is important enough that the dis- 
creteness and omnidirectionality of language is required. 

Voice-Identification Interfaces 

There are many examples in which characters speak to a computer in order 
to gain access to a restricted area, dangerous function, or sensitive content. 
Although voice-authorization technology is available in the real world, it's 
been a mainstay of sci-fi for several decades. The earliest example in the 
survey is in Star Trek (the original series), as Kirk sets the auto-destruct 
sequence for the Enterprise in the episode "Let That Be Your Last Battlefield" 
(Season 3, Episode 15). Some of these systems check for specific spoken data, 
such as a password or code. Others check for a voiceprint, which includes 
some combination of vocal qualities such as timbre, tone, and spectrum. 
These sci-fi interfaces are simple and offer simple confirmation. 

A few other unique voice security interfaces warrant examination. In the 
film Lost in Space, weapons are vocally secured. When John Robinson picks 
up a gun, he speaks the command "Deactivate safety!" to unlock it. Instantly, 
his voiceprint is checked and the command carried out as he begins to fire 
the weapon (Figure 6.5). We learn later that the gun can be disabled for 
certain people— particularly for the untrustworthy Dr. Smith. In another 
scene, Don West locks the room used to imprison Dr. Smith with a simple, 
forceful "Lock!"; we can reasonably assume that the room won't respond to 
Dr. Smith himself should he try to use a voice command to unlock it. 

This system check for voiceprint is common and, in sci-fi, is occasionally 
thwarted by heroes and villains alike to gain access to restricted areas. 
Some use recordings of authorized people, others assemble passphrases 
from recordings of authorized people, and others, since this is sci-fi, are 
simply able to mimic the person's voice directly. For example, in the film 
series X-Men, Mystique is a metamorph capable of transforming into any 
other person. InX2, to gain access to a government computer system, 


Lost in Space (1998). 

Sonic Interfaces 117 

she transforms into a senator's assistant and uses her voice for access. In 
Star Trek: The Next Generation, the android Data is also able to mimic others' 
voices well enough to bypass voice security measures. 


If there are ways to mimic or replicate authorized voices 
and security is important, adding at least one other form of 
identification, such as a retinal scan, typed password, or face 
recognition system, would strengthen security over one form of 
identification alone. 

A humorous example of a problem with a voice-identification system 
comes from the 2009 Star Trek reboot film when Ensign Pavel Chekov 
tries to authenticate himself to the system in his heavy Eastern European 
accent: "Ensign authorization code: nine -five -wictor-wictor-t wo!" He 
gets increasingly frustrated as it repeatedly fails. If this interface were a 
voiceprint, Chekov's accent would be part of the match. This simple joke 
highlights just a bit of the difficulty present systems have parsing spoken 
language and predicts that it won't get much better in 300 years. 


Even within the same language, there is considerable variation 
in pronunciation across different dialects and idiolects. If a sys- 
tem isn't able to recognize common or even individual variation 
in such characteristics as intonation, pronunciation, diction, and 
rhythm, it may inadvertently deny access to those who should 
have it. Designers need to be sure and include this in their 
requirements for such systems. 

Limited-Command Voice Interfaces 

In Blade Runner, Deckard investigates a set of special photographs recovered 
from a hotel room. Though these photographs appear two-dimensional, 
they contain 3D information captured at the moment the image was taken. 
To examine them, Deckard inserts the photos into an "Esper machine," a 
television-like device he has in his living room. It works like this: After the 
picture is inserted, the screen reveals a blue grid, behind which a scan of the 
photo appears. He stares at the image in the grid for a moment and speaks a 
series of instructions that are short and direct: "Enhance, stop . . . Move in, 
stop . . . Pull out, track right, stop . . . Enter and pull back, stop . . . Track 45 
right, center and stop . . . Enhance 34 to 36 . . . Pan right . . . Pull back, stop. 
Wait a minute. Go right" (Figure 6.6). 

118 Chapter 6 


Blade Runner (1982). 

In response, the Esper zooms and pans the photo on the screen and shows 
three sets of numbers at the bottom of the screen: ZM 0000 NS 0000 EW 
0000. The NS and EW — presumably north-south and east-west coordinates, 
respectively — immediately update to his spoken commands. (Attentive 
audience members will note that later numeric commands do not match 
what is shown on screen.) The only time he uses a full sentence that sounds 
somewhat conversational is at the end of the scene: "Give me a hard copy 
right there." In response, the machine prints the part of the image shown 
on the screen. 

This kind of reduced vocabulary makes it much easier for systems to 
respond to voice instructions. They only need to look for specific patterns 
and can ignore many variables and vocal characteristics that might stump 
a system required to respond to grammar or a larger vocabulary. Such 
limited-command voice systems are already common today, and they are 
at the heart of hands-free automobile technology such as OnStar, mobile 
technology such as Apple's Siri, and the oft-maligned voice-response 
telephone support systems. 


The smaller the vocabulary that a voice recognition system 
needs to understand, the more confidently it can identify key- 
words. More constrained contexts help this. The Esper machine 
in Blade Runner, for example, only needs to recognize a few 
commands for spatially exploring a photographic scan. If he 
was far enough from the microphone that only "ack right" reg- 
istered, the system could assume he meant "track right." If you 
are developing a limited-command voice interface, consider 
reducing the glossary that a user must memorize to the small- 
est feasible size. 

Sonic Interfaces 119 


Many current systems are designed in such a way that if you 
add words and phrases that aren't part of its vocabulary, 
the system fails to recognize the words it does understand. 
Although the examples mentioned above focus on the use of 
limited vocabulary, we have to imagine what would happen in 
the case of a character using more than the specific, expected 
vocabulary. For example, if Deckard commanded the Esper ma- 
chine with "Pull out and track right to the cell two over, please," 
we wouldn't expect the system to be stumped. But that's true 
of many nonconversational voice systems today. 

Some users cannot easily distinguish between limited voice-command 
systems and conversational systems, so owing to their social training, they 
speak to these systems conversationally. If the system isn't specifically 
designed to ignore this extra input, in the way Google searches ignore words 
like the, to, and of, the system will attempt to treat the extraneous (though 
polite) content as important. 

One unusual vocal interface found in our survey is the "weirding module" 
seen in the film Dune (Figure 6.7). The Atreides clan uses these vocal 
weapons to amplify specific words and vocalizations, then focus and project 
them as energy. Part of the device is worn around the neck, with two silver 
cylinders placed on either side of the throat, and the other is a box held in the 
hands and used to position the direction of the energy. Paul Atreides quickly 
masters the device, and when he escapes to live with the Fremen, he teaches 
them how to make and use these devices as well. Eventually, they learn that 
his Fremen name, Muad'Dib, is the most powerful vocalization possible 
for the device. In this example, the spoken words are both content and the 
interface for activation. 

Dune (1984). 

120 Chapter 6 


The more obvious the system commands, the easier the inter- 
face will be to learn and use. But be wary of words that have 
multiple meanings or are easily confused within the context of 
use. For example, a self-driving car could be designed to re- 
spond to the words "left" and "right," but without a qualifier like 
"turn" they could inadvertently change course while a political 
discussion ensues between passengers. Even then, the possibil- 
ity of disastrous misinterpretation exists, which is why many 
current systems (like Siri) and sci-fi systems (like the Enterprise's 
computer) use a moniker to signal when the system should take 
notice for a command (see below for more on this). 

Conversational Interfaces 

Truly natural language interfaces still represent the pinnacle of vocal 
interfaces, and the real world hasn't gotten there quite yet. Modern voice 
response systems may seem conversational, but these are actually elaborate 
limited-response interfaces that include polite niceties and respond to key 
words. To be a truly conversational interface, systems need to handle the 
full complexity of conversational language, including unexpected requests 
and responses. 

For example, when J. F. Sebastian returns to his apartment at the top of a 
vacant building in Blade Runner, he is greeted by two handmade automaton 
toys: a clown and a bear (Figure 6.8). When they approach he says to them, 
"Evening fellas," to which they respond "Home again, home again, jiggity 
jig. Goooood evening J. F.!" Their manner conveys that this is a routine 
interaction between them, which might make us suspect they were simply 
checking for his voiceprint. But when they see his guest, Pris, they look 
nervously at her, and he explains to them in casual language that she is 
a friend and to be trusted. This interaction suggests they have greater 
conversational capabilities than was at first apparent. 


Blade Runner (1982). 

Sonic Interfaces 121 

to visit those In die deptlis, 
i order to destroy 

Nearly every mechanical robot, android, and 
gynoid in sci-fi has a fully conversational 
interface. As mentioned above, as early as 
1927 we see the robot Maria in Metropolis 
responding to spoken instructions given to her 
by Joh Frederson (Figure 6.9). 

in whose unage you were creaf 

Metropolis (1927) 

In contrast, artificial intelligences like HAL 
didn't display conversational interfaces 
until at least four decades later. Audiences 
immediately accepted that human-shaped 
machines could speak but could not believe a 
nonhumanoid machine could master language 
until much later. Once the idea was out there, nearly all nonhumanoid 
artificial intelligences were given the ability to speak: Max from Flight of the 
Navigator, Deep Thought from The Hitchhiker's Guide to the Galaxy, even the 
user-friendly, godlike galaxy that Bender encounters while floating in deep 
space in the Futurama episode "Godfellas." 

Computers on Federation ships in the Star Trek films and TV series have 
fully conversational interfaces (voiced by the beloved voice actress Majel 
Barrett Roddenberry across six series, four films, and seven video games). In 
Star Trek, the ship's computer system parses natural speech and responds, in 
kind, with natural (if sometimes slightly formal) language. 

The ship's computer must be addressed with a control phrase to get 
its attention, namely by saying "Computer." Many limited-command 
interfaces work the same way. Perhaps this was so audiences could tell 
when a character is addressing the machine instead of addressing another 
character, but it also helps the system know when it should pay close 
attention and will be expected to understand and respond. Though the 
system is alerted by a control word or phrase, with both HAL and the 
Enterprise computer, the rest of the interface is conversational. This is, of 
course, consistent with conversational dynamics in public or when many 
actors are present. We often need to call out the name of whom we're 
addressing when in conversation around others. 


If users are having conversations that aren't intended for the 
system, there's a risk that the system will try to mistakenly act 
on what it hears. This could be frustrating for users who have 
to retroactively undo some accidental action or wonder why 
the computer is not responding. 


Chapter 6 

Ideally, the system itself should be sophisticated enough to determine when 
commands are intended for it in the same way people do: by monitoring 
the user's gaze, inferring from the content of conversation, and asking 
when it is uncertain. But in cases in which a user needs to be discrete or the 
technology isn't sophisticated enough, give the user controls. 

If such conversations are common, this could be handled with discrete 
modes, such as "listen to me" and "don't listen to me," activated with a 
manual control, a screen control, or voice commands. 

If such conversations with the computer are the exception, a simple escape 
phrase can set the temporary attention for the duration of the conversation. 
People are social animals and are used to addressing each other by name, 
so a good tactic is to let the user set a temporary "listen to me" mode by 
addressing it by name. A common way this is done is for the system to listen 
for an alert word or phrase, such as its name, as HAL does in 2001: A Space 
Odyssey, or a more generic term, like the word computer in Star Trek. 

Alternately, if conversations with people are the exception, the system can 
be designed such that when the user begins a statement with a person's 
name to disregard it. This way if Picard needs to say, "Data, shut down," he 
doesn't accidentally disable the entire ship. 

Once a Star Trek crew member gets the computer's attention, the 
conversational interface mirrors human conversation greatly. It can 
parse complex grammar, nuanced word senses, commands split between 
users, colloquialisms, and cultural references. It understands hundreds of 
languages and dialects and can shift seamlessly between them. This kind of 
sophistication makes for an exceedingly powerful and natural interface, but 
it is difficult (some have argued, impossible) to build, and expectations for its 
use are as high as they would be for a human. 


If a system seems to have full conversational capability, users' 
expectations will be that it can follow basic human rules of 
conversation. These include acceptable pacing, the role of po- 
liteness and honorifics, speech disfluencies such as "umm" that 
signal to the listener that they should hold on while the speaker 
is thinking, rules for interruption, nonlexical conversational 
sounds and discourse markers such as "mm-hmm" and "wow" 
that confirm that the listener is still engaged and in agreement, 
and the four so-called Gricean maxims. 1 

1 These four maxims are quantity (speakers provide as much information as necessary but 
not too much), quality (speakers are truthful), relation (speakers don't stray too far from the 
subject), and manner (speakers are clear and brief). 

Sonic Interfaces 123 

Advanced conversational interfaces need to be aware of these sorts of social 
rules, understand them, and respond in socially appropriate ways in order to 
fulfill their role as considerate, if subservient, conversationalists. 

Sonic Interfaces: Hearing Is Believing 

Sonic interfaces seem less sci-fi than many other technologies we explore in 
this book because we commonly encounter many types of sonic interfaces 
in our lives today, such as system beeps and limited-voice interfaces. Users 
may prefer and expect the naturalness of truly conversational interfaces, 
but these are still difficult to build convincingly outside of sci-fi. User 
expectations of successful sonic interfaces increase significantly across the 
spectrum described here. 

Simple system sounds need to be clearly heard and understood, but we 
don't have greater expectations for them. The more a voice is present in 
the interface, the more users expect that voice to represent the social 
characteristics of another person. This is especially true of the human quality 
of the voice used (the more representational, the higher the expectations). 
There aren't many lessons at either end of this spectrum because the first are 
so simple and the last use human conversation as their model. 

As Marshall McLuhan famously observed, "We are simply not equipped 
with earlids." We evolved to constantly monitor sounds as evidence of the 
truth of the things around us, and to pay attention to them for meaning. 
Sci-fi has long recognized this characteristic, harnessing it to make its 
speculative technologies believable, useful, and inspirational. Almost from 
the beginning, sci-fi has filled our ears with the promise of technology with 
which we could fully converse, using the full power that language brings. 
Though we're still a long way away, we can listen carefully for those things 
that work and make our technology sing. 

124 Chapter 6 


Brain Interfaces 

Physically Accessing the Brain 126 

Disabling the Mind 131 

Two Directions of Information 132 

Active Subjects 144 

Dismantling Two Sci-Fi Brain-Tech Myths 151 

Where Are the Thought Interfaces? 153 

Brain Interfaces: A Minefield of Myths 155 


Buck Rogers, "War of 

the Planets" (c. 1939). 

Buck tightens the chinstrap on the tall mind-control helmet placed 
on Killer Kane's head (Figure 7.1). Kane's face goes blank and 
expressionless. Buck says, "You'll take orders from me now, Kane," 
before walking him to the space radio and instructing him to broadcast the 
command, "This is the Leader Kane. Withdraw all outer atmosphere patrols 
to their flying fields," thereby ending the War of the Planets. 

Thought is invisible. And the thing that houses our thoughts, the brain, is 
hidden in the protective bone box of our skulls. Even if we could see our 
brain when our thoughts are racing at 100 miles an hour, it would look like 
it's just sitting there. But audiences and sci-fi makers alike are aware of the 
importance of this organ as the seat of thought, and of the central role that 
thought and memory plays in human life. This proves an irresistible lure 
for the imagination of speculative technologies, but it also poses a creative 
challenge to sci-fi makers: How do you make interactions with this most 
invisible and unmoving of materials apparent to the audience? 

Physically Accessing the Brain 

To indicate to an audience that a device is accessing a user's thoughts, 
designers first have to show that the technology has access to the brain. 

Invasive Brain Interfaces 

The most direct way is to show technology plugged into a person's brain 
or nervous system, but this sort of invasive connection naturally triggers 
negative reactions from most people. For most of us, seeing a foreign object 
sticking out of a person's head or neck indicates a dire medical emergency or, 
at the very least, something seriously wrong. Though some sci-fi makers seek 
to capitalize on this sort of body horror, most do not. 

126 Chapter 7 

FIGURE 7.2a,b 
The Matrix (1999). 

Dollhouse, "Epitaph: 
Return" (Season 2, 
Episode 13, 2010). 

In The Matrix films, for example, a menacing and sharp plug is inserted into 
the base of the skull to allow that person to "enter" the Matrix (Figure 7.2). 
In another example, the final episodes of Dollhouse showed jacks implanted 
in the face that let people upload and erase information and skill modules at 
will (Figure 7.3). 

Despite these two examples, direct-connection brain interfaces are rare. 
Most interfaces in sci-fi don't rely on technologies that pierce the skin. 

Noninvasive Brain Interfaces 

The easiest noninvasive way to show access to the brain is by proximity. 
Most audiences assume that technology put near or around someone's 
head probably has something to do with the brain. This definition of "head" 
includes the forehead but usually excludes the face, perhaps because things 
near the face might be interpreted as having more to do with either speech 
or the sensory organs— eyes, ears, nose, and mouth. The face might be 
excluded from these kinds of connections in sci-fi simply so audiences can 
see the actor's expressions, but the face is a preventatively thick barrier 
between sensors and the brain anyway. 

There are other noninvasive ways to show the connection between the 
brain and technology, but the need for proximity is closest to the actual 
science. Thoughts are complicated and ill-defined things, and what a real- 
world brain-computer interface technology actually measures is the faint 

Brain Interfaces 


electromagnetic waves emanating from the brain as a result of clusters 
of neurons firing. Because many of the brain's regions are specialized for 
certain types of thought, scientists must scatter many sensors around the 
skull to pinpoint the region sending a particular signal. Audiences and 
sci-fi makers have likely seen images of skullcaps with lots of small sensors 
and wires emanating from them, providing a real-world paradigm for 
speculative brain-computer interfaces. 

Worn Devices 

Noninvasive direct-contact brain interfaces in sci-fi take two main forms: 
smaller devices that fit on the head, and larger machines that the user sits in. 

Most brain interfaces seen in the survey are devices that are worn on the 
head — especially the crown (Figure 7.4). In Metropolis, the mad scientist 
Rotwang's machine uses a skullcap. In the Buck Rogers serial of 1939, Killer 
Kane controls his worker zombies by means of a tall metallic hat (see 
Figure 7.1). In Brainstorm and Johnny Mnemonic, the devices fit around 
the user's head. The Lawnmower Man straps a helmet onto himself and his 
victims. In Flight of the Navigator, David wears a metallic headband. The 
Game from the Star Trek: The Next Generation episode of the same name 
has players wearing the devices like glasses. The mind-blanking haloes in 
Minority Report are worn like earphones. 

FIGURE 7.4a-g 

Metropolis (1927); Brainstorm (1983); 
Johnny Mnemonic (1995); Lawnmower 
Man (1992); Flight of the Navigator 
(1986); Star Trek: The Next Generation 
(1991); Minority Report (2002). 


Chapter 7 

Other devices are larger and require users either to sit inside or to strap 
themselves into them. The Krell thought-manifesting technology in 
Forbidden Planet requires users to lean forward and lower two long rods to 
their temples (Figure 7.5a). Both deneuralizers from Men in Black 2 are large 
and require K to sit within them (Figure 7.5b, c). Agents working for the 
Dollhouse have their minds wiped and replaced while sitting in a reclining 
chair (Figure 7.5d). Tom Paris from Star Trek: Voyager flies an alien shuttle 
with an artificial intelligence named Alice from a seat with an affixed 
"neurogenic" interface (Figure 7.5e). 

FIGURE 7.5a-e 

Forbidden Planet (1956); Men in Black 
2 (2002); Men in Black 2 (2002); 
Dollhouse, "Ghost" (Season 1, Episode 
2, 2009); Star Trek: Voyager, "Alice" 
(Season 6, Episode 5, 1999). 

Brain Interfaces 129 

Where Do Sci-Fi Interfaces Sit? 

Where do sci-fi audiences expect brain interfaces to sit? 

When we overlap all of the brain interfaces from the survey onto a single head (Figure 7.6), 
we can see an image of the common physical location of brain interfaces. An arc over the 
head from ear to ear is there for stability, but it also floats away from the head when the 
subject is reclining. Sagittal arcs play a strictly stabilizing role for technology. 

The crown is the most important location for these technologies. The forehead appears 
in cerebral sci-fi such as Star Trek and Forbidden Planet. More visceral sci-fi, such as The 
Matrix, focuses on the back of the head. Whole-head and cranial interfaces turn out to be 
closer to real-world science, which needs input from all areas of the brain. 

Though the industrial design of real-world brain interfaces relies on the particular science 
involved, industrial designers should be aware of the expectations that have been set. 

FIGURE 7.6a,b 

Overlaying physical brain interfaces from sci-fi reveals some patterns. 

Remote Connection 

Two examples show brain access from a distance, with victims usually 
unaware they are being manipulated. 

In the final episodes otDollhouse, the doll technology has advanced to a 
stage where any person's mind can be wiped from a distance (Figure 7.7). 
This throws all of civilization into terrified, anti-technology chaos. 

In the Star Trek: The Next Generation episode "The Battle," Ferengi commander 
DaiMon Bok uses a highly illegal and rare Thought Maker to exact a 
complicated revenge on Commander Picard. The Thought Maker is a large 
broadcast device for painfully implanting false memories in a target. It is 
paired with a smaller receiver device that produces even stronger effects when 
near its victim (Figure 7.8). (More on this below, in "Unwilling Subjects.") 


Chapter 7 

FIGURE 7.7a,b 

Dollhouse, "The Hollow Men" (Season 2, Episode 12, 2010). 


Star Trek: The Next Generation, 

"The Battle" (Season 1, Episode 9, 1987). 

Disabling the Mind 

The survey includes a few technologies whose purpose is to disable the mind 
of the subject: to forbid its subject any agency, thought, or the development 
of new memories. These interfaces are particular to their properties, and few 
generalities can be made other than their proximity to the brain. 

In Buck Rogers, this technology comes in the form of a metallic top hat with 
spiraling coils and dials. In Dollhouse, the same reclining chair that uploads 
experiences can be used to wipe an uploaded mind and return the doll to 
a harmless, dopey state. In Minority Report, it is a headband with a single 
glowing indicator at the rear (Figure 7.9). 

FIGURE 7.9a,b 
Minority Report (2002). 

Brain Interfaces 


Two Directions of Information 

We can also distinguish brain interfaces as addressing two different 
directions of information. Interfaces that enable readingfrom the brain treat 
a person's thoughts and memories as output. Interfaces that enable writing 
to the brain incorporate the user's thinking as an input. 

Writing to the Brain 

Some interfaces are meant to install new information into a subject's brain. 
The technology takes a much different form if the subject is unwilling, so 
we've separated the examples along these lines. 

A note about triggers: It's tempting to include interfaces that trigger 
preprogrammed effects in a subject's mind, such as in The Manchurian 
Candidate, where seeing a queen of diamonds playing card would turn Shaw 
into a sleeper agent for the KGB, or the Dollhouse trigger whispered over 
the phone to Mellie ("There are three flowers in a vase. The third flower is 
green"). Although the trigger signal may be delivered through any media, 
such as a mobile phone, there are no specialized interfaces seen in the 
survey for delivering these signals, so we don't include them here. 

Willing Subjects 

If the subject is willing (or at least unresisting, as are the dolls in 
Dollhouse), the interface is most often a reclining chair with head-mounted 

The technology in Until the End of the World was originally intended by its 
creator to give sight to the blind through direct brain stimulation. To see 
the images, the blind subject rests in a reclining chair with her head in a 
horseshoe-shaped band of blue lights and an electrode strapped to her head 
(Figure 7.10a). The device that maps the mind of Joel horn Eternal Sunshine 
of the Spotless Mind has a similar hard plastic halo around the head, but the 
subject is sitting upright as his mind is read and mapped (Figure 7.10b). 

FIGURE 7.10a,b 

Until the End of the World (1991); Eternal Sunshine of the Spot/ess Mind (2004). 

132 Chapter 7 


FIGURE 7.11a-d 
The Matrix (1999). 

Perhaps the best-known example of this type of brain interface is seen in 
The Matrix. Whether to access the Matrix or to train for operations, Neo and 
the others are "jacked in" to specialized systems. For training, skill modules 
such as kung fu are uploaded into their brains. The operator of this system 
sees animated visual icons on a control screen that show the skills module 
and a brain "filling up" with knowledge, serving as a rotating, 3D-rendered 
progress bar of sorts. Neo is able to immediately put these skills to the test 
inside the system's virtual reality (Figure 7.11). 

This is similar to the Dollhouse procedure in which new personalities and 
memories are uploaded into human "dolls," but no process is shown that 
an operator might see. Instead we are shown a stylistic representation of 
the new memories rushing toward the camera as a field of images floating 
against a black background (Figure 7.12). Avatar uses similar industrial 
design and rushing visualizations for its similar technology (see Figure 5.4). 

FIGURE 7.12a,b 

Dollhouse, "Spy in the House of Love" (Season 1, Episode 9, 2009). 

Brain Interfaces 


FIGURE 7.13a-c 
Chrysalis (2007). 

Another mind-writing device appears in the French sci-fi thriller Chrysalis. 
The device is meant to "shape memories," but a gangster is using it to 
implant memories. Though his abuse of the technology is nefarious, the 
unaware subjects sit in it willingly (Figure 7.13). 

The central plot device of Johnny Mnemonic involves a technology that 
modifies a human brain to act as a simple repository for smuggling data 
over borders, while it remains inaccessible to the carrier. Uploading 
data is a process that involves plugging a small cable directly into the 
skull and donning some protective gear to avoid damage during the very 
painful procedure. One imagines that a reclining position would be more 
comfortable, but Johnny has a professional, tough-guy reputation to 
uphold (Figure 7.14). 

FIGURE 7.14a,b 
Johnny Mnemonic (1995). 


Chapter 7 


Some sci-fi treats brain-writing technology like an MRI scan, 
where the procedure is interrupted by a subject's moving. If 
this is the way actual noncontact brain-writing technology 
would work (see below for why this technology is unlikely), 
then providing a comfortable resting position would help to 
avoid movement due to fatigue or discomfort. 

Unwilling Subjects 

When the subject is unwilling, the form of the technology becomes 
particular to the property. 

In both of the TV series Chuck and StarGate SG-1, unwilling victims are 
force-fed vast amounts of data in a short time. In Chuck, the title character 
is stunned into submission by a rapid-fire set of specially encoded images 
displayed on his home computer, embedding in his head the only copy of a 
vast set of US government secrets and rendering him unconscious after it 
is complete (Figure 7.15a). In the StarGate SG-1 episode "Lost City," Colonel 
O'Neil is grabbed by a piece of fluid architecture, and alien information 
is painfully forced into his brain (Figure 7.15b). These examples have fast 
interfaces that are not apparent until they activate (to catch unsuspecting 
subjects off guard). 

The Star Trek: The Next Generation episode "The Battle" revolves around the 
Thought Maker, a highly illegal broadcasting device for implanting thoughts 
into a victim's brain, even to the extent of making him or her disregard 
reality and instead relive events in the past. The Thought Maker itself 
looks something like a large, metallic brain, with a glowing red, bisected 
hemisphere on top and rectilinear, metallic shapes below. The similar- 
looking, head-size receiving device amplifies the Thought Maker's effects 
when it is near its victim (Figure 7.16). 

FIGURE 7.15a,b 

Chuck, "Chuck Versus the Intersect" (Season 1, Episode 1, 2007); 

StarGate SG-1, "Lost City" (Season 7, Episode 21, 1997). 

Brain Interfaces 


FIGURE 7.16 
Star Trek: The Next 
Generation, "The 
Battle" (Season 1, 
Episode 9, 1987). 

To use the Thought Maker, DaiMon sets it on a table and rotates the red 
hemisphere along its equator. He also moves a slider backward along its 
equator, but the effect of this control is not apparent (see Figure 7.8). The 
farther the entire hemisphere is rotated, the more pain Jean-Luc endures 
and the more suggestible he becomes. DaiMon also has a screen with which 
he monitors the device's effects (Figure 7.17a). Its mysterious variables are 
displayed as irregular blue bars that angle away from a horizontal axis. The 
more intensely the device is working, the faster a dark-blue banding slides 
along the bars and the louder its whine (Figure 7.17b). 

FIGURE 7.17a,b 

Star Trek: The Next Generation, "The Battle" (Season 1, Episode 9, 1987). 


Chapter 7 

A few technologies write to the brain by deleting memories, just as a computer 
might delete files by overwriting them. The neuralyzer from Men in Black is a 
handheld rod that flashes a bright red light toward its subject. People viewing 
this light without the protection of special, stylish eyewear immediately 
forget recent events and pass into a brief fugue state, during which they are 
subject to false memory suggestion through speech. The interface— one of 
the few user-facing brain interfaces seen closely in the survey— is a set of 
dials for selecting the duration of memories to erase, some lights to confirm 
selection and power, and a push button to execute (Figure 7.18). The devices 
used to alter memories in The Adjustment Bureau are also handheld cylinders, 
but one shines lines of light into the subject's eyes, and the other ends in an 
illuminated disk held near the subject's temple (Figure 7.19). 

FIGURE 7.18a,b 
Men in Black (1997). 

FIGURE 7.19 

The Adjustment Bureau (2011). 

Brain Interfaces 


FIGURE 7.20 
Paycheck (2003). 

FIGURE 7.21 

Eternal Sunshine of the 

Spotless Mind (2004). 

Other interfaces for wiping memories look a lot like devices from Dollhouse 
and Until the End of the World: glowing arcs across the coronal plane of a 
resting subject that are connected to computers (Figure 7.20). 

The device that does the memory erasing in Eternal Sunshine of the Spotless 
Mind works in the subject's sleep while a technician monitors the process. 
(This is different than the device in Figure 7.10b, which maps the mind.) The 
device itself looks like a large metal bowl covering the top of the head with 
wires regularly spaced across it (Figure 7.21). 

Reading from the Brain 

Some brain-affecting interfaces extract information passively from their 
subjects. For these interfaces, a distinction between willing and unwilling 
subjects isn't particularly useful. For these speculative interfaces, the main 
challenge for designers is to show that the machines are working, what has 
been extracted, and when the process is complete. 


Chapter 7 

In Metropolis, Rotwang copies Maria's mind into a robot. As the extraction 
occurs, electricity arcs between Maria's chamber and a sphere directly 
above it, and strange chemicals bubble in flasks. The helmet she wears in 
the chamber is a material conduit and doesn't provide any signal. We know 
the data— in this case, her mind— has been copied when we see the robot's 
stiff, metallic appearance transform until it looks and moves just like Maria 
(Figure 7.22). 

FIGURE 7.22a-f 
Metropolis (1927). 

Brain Interfaces 




FIGURE 7.23a-c 

Flight of the Navigator (1986). 

In Flight of the Navigator, David has mysteriously had alien information, such 
as star charts and ship schematics, deposited into his brain. Luckily, the 
scientists at NASA can pipe this data directly out to displays as pixel-perfect, 
animated graphics via a brain interface strapped around his forehead 
(Figure 7.23). 

In Minority Report, three psychically gifted triplets are kept sedated in a 
pool, and their precognitive visions are piped directly from their minds to 
video systems that display and record them. The audience knows the system 
is working because the video screens mounted in the ceiling directly above 
them are flashing images, and the lights in their headgear and in the pool 
are illuminated (Figure 7.24). 

The Dollhouse technology allows the mind of the subject to be read and 
a copy of it stored digitally, so that a person's memories and skills can be 
uploaded to a "doll" later. The procedure requires the subject to lie back on 
a reclining chair with his or her head resting within a horseshoe-shaped 
component (see Figure 7.12a). An operator starts the copying via a nearby 
computer or by a few controls on the component itself. We know the device 
is on and working primarily by the blue glow that surrounds the head of the 
subject. We know it is complete when the light dims and the chair raises the 
subject back to a sitting position. 


Chapter 7 

FIGURE 7.24a-c 
Minority Report (2002). 

Wim Wender's epic Until the End of the World features a telexperience 
technology that is adapted to record dreams so that they can be shared with 
others or watched on a video display. In the world of the film, the technology 
is new and experimental, and is handled in the inventor's laboratory with 
complicated electronic equipment. After having first recorded an event with 
a special binocular, brainwave-reading camera (Figure 7.25a, b), the subject 
must rewatch the recording and remember the event (Figure 7.25c, d). 
During this rewatching, the subject's brainwaves are being recorded for 
later transmission. The subject has electrodes taped to his or her temples 
and lies back in a curved headrest with an electronic grid lining it. We know 
it's working because of the various screens that display the moving images 
gathered by the procedure. We know it's done when the subject's REM state 
ends, which is seen on screens that monitor the health and brain activity 
of the subject. 

Brain Interfaces 141 

FIGURE 7.25a-d 

Until the End of the World (1991). 


Interfaces that take time to read from a source should signal 
when they are in process and when they are complete. Sci-fi 
primarily uses two means to visualize the process. In the first, 
real-time results of a reading appear on a nearby monitor. This 
nicely shows what content has been detected, and can even 
help signal the amount of progress for a lengthy scan. In the 
second case, where the content is too complex to display as 
a visual, lights on the reading device near the subject's head 
illuminate and animate when the read is in progress. Either or 
both strategies communicate well to observers. The animated 
icons in various operating systems that loop during long copy 
procedures are one example, though few show content and 
most could be simpler. 


Three sci-fi films in the survey tackle the personal and cultural effects of 
technology that records someone's sensory experiences and allows someone 
else to fully experience them as if they were there, passively experiencing 
it themselves. This sort of technology is categorized somewhere between 
reading and writing to the brain, because recording the experience is 
separated in time from its later playback. 


Chapter 7 

The first film, Until the End of the World, is discussed for its brain-reading 
capabilities above. The playback process is simpler. Initially the (blind) 
viewer lies back with her head in a ring of pulsing blue lights. Later in the 
film, when the technology is adapted to record and play back dreams, the 
resulting images can be viewed on any video screen like a movie, though it is 
strictly a visual, not a full-sensory, experience (Figure 7.26). 

The second of these films is Brainstorm, which focuses on the personal 
and corporate abuses possible with such a technology. It also shows the 
iterations that the technology goes through— from clunky laboratory 
prototype to slick consumer accessory (Figure 7.27). 


r m 



W i 



r M 


FIGURE 7.26a-d 

Until the End of the World (1991). 

FIGURE 7.27a-c 
Brainstorm (1983). 

Brain Interfaces 


FIGURE 7.28 
Strange Days (1995). 

The most recent of these films is Strange Days, in which a peddler of virtual- 
reality pornography tries to catch a murderer who uses the technology in 
disturbingly twisted ways. In the story, the medium itself is suspect and 
regarded by some characters as entertainment fit only for lowlifes. So it is no 
surprise that the device has no external indication that it is on and is most 
often worn under wigs (Figure 7.28). 

Active Subjects 

The other major category of brain interfaces lets a willing user control a 
system with thought as the primary input. Though it seems like sci-fi would 
be all over this, a look through the survey shows it is comparatively rare. 
There are a few telepresence technologies, and then a handful of one-off 
technologies that are particular to their plots. 

Unlike the passive telexperience technologies discussed above, telepresence 
technologies involve an active participant who is able to act and influence 
the experience he or she is having. 

Virtual Telepresence 

Two properties in the survey address technologies that allow users full- 
body immersion into virtual worlds. The first is one of the best-known sci-fi 
trilogies of all time, The Matrix. 

By jacking in through the socket in the back of his head, Neo makes himself 
available to an operator who can transfer Neo's consciousness into the 
virtual reality of the Matrix. Neo, through his avatar, interacts in the Matrix 
with full agency as if it were a real world with full sensory stimulus and 
tactile feedback (Figure 7.29). 

The rebellion's understanding of the digital nature of the virtual world allows 
them to use thought as input; to bend the rules of the simulation, performing 
seemingly superhuman feats to the amazement of people still trapped in the 
Matrix, who are unaware of its true nature and bound to its laws. While a 
rebel's consciousness resides in the Matrix, he or she is effectively in a coma in 
the real world, insensate and unresponsive (Figure 7.30). 

144 Chapter 7 

FIGURE 7.29a-d 
The Matrix 0999). 

FIGURE 7.30 

The Matrix Reloaded (2003). 

Members of the rebellion have even created virtual spaces with the sole 
purpose of facilitating specialized tasks, such as training for combat or 
managing security access to the underground human city of Zion. The 
virtual interface in this environment does not need to obey any real-world 
rules (see Figure 5.8d). 


The more that virtual spaces deviate from the real world, the 
more foreign they feel to the people inhabiting them. Our 
brains and bodies are fairly well adapted to living in the real 
world, and we have many faculties hardwired into the brain 
to help us: face recognition, spatial hearing, simple physics 

Brain Interfaces 


algorithms, and an aptitude for spoken language, to name a 
few examples. Spaces that deviate greatly fail to take advan- 
tage of these built-in faculties and require more cognitive ef- 
fort to make sense of and operate within. 

Like with all media, however, adhering to real-world metaphors 
when the functionality doesn't align can also create confusion. 
Don't be afraid to deviate from reality when the functions don't 
fit how things work in reality. 

In the Deep Space 9 episode "Extreme Measures," the effects of Bashir and 
O'Brien's improvised mind probe are shown cinematically— that is, the two 
of them are shown walking around in a spaceship looking for a physical file 
when, in fact, they are probing the mind of an unconscious enemy to look for 
vital information (Figure 7.31). When they find the "file" and wake up in the 
real world, we know their task is done. Note, however, that in this episode, 
they "wake up" several times but are still in Sloan's mind, tricked into 
thinking they're now back in the real world. This brings up both a common 
sci-fi theme with brain interfaces as well as a practical challenge when 
creating these virtual worlds: How do people know when they're out of the 
mind-created world and back in the real world? 

In the film eXistenZ, a game system immerses players in a virtual reality 
where they play roles in an exciting story. In a recursive twist, the virtual- 
reality game involves a gaming system similar to the real-world system, but 
its appearance is more biological than technological (Figure 7.32). 

FIGURE 7.31a,b 

Star Trek: Deep Space Nine, "Extreme Measures" (Season 7, Episode 23, 1999). 


Chapter 7 

FIGURE 7.32a, b 

The real-world system uses headgear and a controller in the player's hands 
(Figure 7.33a). The in-game biological system has similar controls for the 
player's hands, but the other component is a spinal jack with a cord that 
looks disturbingly like an umbilical cord (Figure 7.33b, c). 

This film provides the most direct comparison between two versions of the 
same speculative technology, one electronic and the other bioengineered. 
Functionally they're identical, but the biological version feels creepier — even 
the noninvasive handheld controller. Should such technologies become 
commonplace, they might not inspire such a visceral reaction, but until 
then, designers should be aware of how unnerving such interfaces can be. 

Brain Interfaces 



Biology has a distinct look and behavior guided by evolution- 
ary rules, especially when compared to man-made technology. 
When the aesthetics and movements of the body appear in 
ways that violate our understanding and experience of biology, 
we feel revulsion, a violation of our shared and near-universal 
biophilia. This effect is heightened when it appears that this 
"natural" material has been subverted for man-made ends. 
Designers should be aware of this visceral response to viscera, 
and either make direct use of it for creepy ends, or take pains 
to avoid it for any other purpose. 

Actual Telepresence 

Avatar's technology allows a subject's consciousness to be projected to a 
purpose-grown humanoid creature. The interface is an enclosed bed with 
a side-hinged chamber for body-monitoring sensors. The only head-related 
sensors are a few thin transparent lines with bright pinpoints that arc 
around the subject's head at a distance of a few inches. While the mind is 
being projected, the human body remains in a disabled state, protected in 
the coffin-like bed, which rests in turn inside a large metal cylinder similar 
to a computed tomography (CT) scanner (Figure 7.34). 

FIGURE 7.34a-c 
Avatar (2009). 


Chapter 7 

FIGURE 7.35a-e 
Forbidden Planet (1956). 

Manifesting Thought 

In Forbidden Planet, Dr. Morbius demonstrates the oddly named "plastic 
educator," a Krell device that manifests any thought the user has. (It doesn't 
appear to actually educate.) To use it, he sits in a chair and swivels three 
lit rods to touch his head, and then concentrates. As the thought becomes 
clearer, its manifestation resolves. Once manifested, the translucent 
projection moves of its own accord as the conceived object would. Morbius 
manifests an image of his daughter, who stands, smiles, and shifts her 
posture (Figure 7.35). To turn off the device and dismiss the manifestation, 
he lifts the rods to their original position. 

Having Virtual Sex 

When Lenina makes an offer to John Spartan for "sex" in Demolition Man, it 
turns out to be a meeting of the pleasure centers of their minds. To perform 
the act, they sit a few feet apart and don one of a pair of headgear, then close 
their eyes and relax. The headgear has a small red light to indicate that it is 
on. Each feels pleasure and sees psychedelic, sensual images of the other. 
Disconcerted with the unfamiliar, virtual nature of the experience, Spartan 
removes the headgear before climax, so there's no indication of how the 
interaction ordinarily culminates (Figure 7.36). (See Chapter 13 for more on 
interfaces in this genre.) 

Brain Interfaces 


FIGURE 7.36a-c 
Demolition Man (1993). 

Piloting a Spaceship 

In the Star Trek: Voyager episode "Alice," Tom Paris is selected by the artificial 
intelligence in an alien shuttle to be its "savior" of sorts. It projects an image 
of a human woman into his mind and, through this hallucination, convinces 
him that he must pilot the shuttle to a specific destination. To do so he sits in 
the pilot's seat and rests his head in a "neurogenic" interface. It is through this 
interface that he flies the shuttle. The audience knows that the neurogenic 
interface is on and functioning when green pinpoint lights along its surface 
illuminate and its retractable arm extends horizontally across the pilot's 
forehead. Paris has access to a touch-screen interface on the instrument 
panel, but all dialogue with the artificial intelligence is conducted through 
speech, and the piloting through the brain interface. After he has been piloting 
the shuttle for some time, additional glowing wires appear to grow from the 
chair, snaking across his body and lodging themselves in his skin, illustrating 
the AI's increased absorption of him (Figure 7.37). 

FIGURE 7.37 
Star Trek: Voyager, 
"Alice" (Season 6, 
Episode 5, 2000). 

150 Chapter 7 

FIGURE 7.38a,b 

Star Trek: The Next Generation, "The Game" (Season 5, Episode 6, 1991). 

Playing a Game 

To introduce Riker to The Game, Etana Jol places a small headset on his 
crown with thin transparent arms that wrap around in front of his eyes. To 
turn it on, she presses small controls on the headset above his ears. Small 
red beams emanate from the tip of the arms to his pupils, projecting the 
graphics of the game directly onto his retinas (Figure 7.38). The device reads 
brainwaves, allowing the player to control the 3D position of a small virtual 
disk. When the player moves the disk into a funnel, the level is complete, and 
he receives a small buzz to the pleasure center of his brain. The game turns 
out to be addictive, mind-altering, and somewhat infectious, as it programs 
players to convince others to play the game as well. 

Dismantling Two Sci-Fi Brain-Tech Myths 

Because the brain-affecting interfaces discussed above are either based 
on bad science or so unlike any other technology that interface designers 
are likely to be working with, direct lessons aren't apparent. Instead, let's 
identify the two problematic concepts that become evident when looking at 
the entire collection. 

Myth: Brain-Affecting Interfaces Will Be Painful 

In most of these properties, moving information in or out of the brain is 
painful to the subject, even when the technology is noninvasive. Subjects' 
heads are immobilized and their body reclined to a resting position, as if 
to minimize potential damage and discomfort. This frame is problematic 
because it isn't true for today's real-world, state-of-the-art technology, and 
isn't likely to be true in the future, either. 

Brain Interfaces 


In the real world, the closest science has come to putting data directly 
into the brain is with a procedure called transcranial magnetic stimulation 
(TMI). At best, it can keep subjects alert and aware, and Australian TMI 
researcher Allan Snyder has shown it can improve subject's scores at math 
tasks. Otherwise it can dim localized faculties such as inhibition, cause 
involuntary jerking of muscles, or cause the subject to see white spots 
of light in their vision. At worst, it can cause seizures, but it doesn't give 
subjects bodice-ripping migraines like sci-fi would have us believe. Nor has 
it deposited any "information" into anyone's brain, and it is unlikely to (see 
the next section). 

The closest we've gotten to getting data directly out of the brain is with 
functional magnetic resonance imaging (fMRI) neuron reading. And as of 
2008, it was strictly a crude, 10x10 pixel copy of the visual field, certainly not 
a rich experience or a whole mind (Figure 7.39). 1 Furthermore, the process 
isn't painful. 

Figure 7.39 illustrates the results of this type of reading. It shows what was 
read off of two subjects. The top row shows what the subjects were being 
shown while in the fMRI machine. The rest shows what the scientists were 
able to extract over multiple readings. The last row shows an average, which 
matches the original most closely. 

contrast pattern 

contrast pattern 

DOBQQ □□□□□□ DOBQQ □□□!!□□ 

Mean of 
contrast pattern 







eiqx2q aaaaaa 
□eseq a BE an a 


FIGURE 7.39 

Scientists have so far only managed to read crude fMRI images from the brain. 

1 Miyawaki, Y., et al. (2008). Visual image reconstruction from human brain activity using a 
combination of multiscale local image decoders. Neuron, 60, 915-29. 


Chapter 7 

Myth: Knowledge Can Be Installed 
and Uninstalled Like Software 

Some sci-fi properties show skills and information being plugged in to 
the hard drive of the brain (either through a jack or through the eyes) and 
uploaded. Just wait for the painful progress bar, and poof! — you know kung fu. 

This metaphor is problematic because it runs counter to modern brain 
science, which says that the structure of the brain is the knowledge of the 
brain. Furthermore, this structure is holographic, meaning bits of information 
related to a single thought are distributed throughout the brain, and the 
neurons involved are also used for other thoughts. To "upload" information 
requires a precise physical restructuring of some significant portion of the 100 
billion nerve cells in the gray matter. Neither flashes of light blasting the retina 
nor electrical impulses shooting through a jack will do it. 

Where Are the Thought Interfaces? 

The main thing missing when we look at the brain-reading technologies is 
thought interfaces. What do we mean by this? Aren't they all thought interfaces? 

Certainly there are physical interfaces for connecting a thinker's brain 
to a machine, but beyond that, there are precious few audiovisual 
representations and controls of the systems shown. In most cases, these 
technologies are really narrative tools that transport the characters to other 
worlds where the characters interact bodily with their surroundings and 
linguistically with others. The workers in Zion, for example, use floating 
gestural interfaces with near-perfect representations of their own bodies 
(see Figure 4.19a). 

In Star Trek: Voyager, Paris seems to be flying the Alice shuttle as an 
extension of himself. We never see the piloting interface that is feeding him 
information and allowing him to modify the ship's position and systems 
accordingly (see Figure 7.37). 

The primary function of Forbidden Planet's plastic educator is clearly to read 
from the mind, but the thinking is the entire interaction (see Figure 7.35). 
When you visualize an object in the way that the machine expects, then you 
see an image of it. That's it. 

The brain-reading technology in Star Trek: The Next Generations game is 
closest, but pretty rudimentary: relax to move the red puck into the blue 
funnel. The interface is also very basic, but there's a feedback loop there to 
accomplish a task and receive a reward, which pretty quickly becomes the 
player's goal. 

Brain Interfaces 153 



Given the kinds of thoughts one can have, physical position 
seems to be one of the most basic as a candidate for specula- 
tive brain-reading technologies. Where are the thinking tools 
for managing power in a starship? Or aiming weapons? Or 
developing and testing a complex hypothesis? They just aren't 
there. These more advanced uses of brain-computer interfaces 
would require complex interfaces, but not the simple ones seen 
in the survey. 

Why the dearth of interfaces, given even a medium-size collection of brain 
technologies? Our guess is that it's too soon on the technology curve, 
and brain-interface technology is too young. As of this writing there 
are only six consumer brain- computer interfaces on the market. Two 
of these are toys that measure beta emissions from the brain with dry 
electroencephalography (EEG) sensors, which correlate to concentration 
and relaxation and actuate a fan to move a ball (Figure 7.40). 

FIGURE 7.40a,b 

Force Trainer; Mindflex game 


Chapter 7 

FIGURE 7.41a,b 
Emotiv EPOC. 

A more advanced and costly option is the Emotiv EPOC system. It is meant 
to be a general input device, like a mouse. Its 14 sensors combine with 
gyroscopes, and it is touted as having, after a bit of training, the ability to 
capture four mental states, certain thoughts and facial expressions, and 
head motion (Figure 7.41). 

Still, even this most advanced of consumer brain-computer interfaces 
doesn't have general market awareness, application, or success. Most people 
don't know about it. This means that there is no established paradigm in the 
real world for sci-fi makers to build on for their speculative technologies. 
With little incentive for sci-fi makers to include these invisible interactions 
in their moving pictures, reality will have to lead the way in establishing this 
paradigm before we can expect to see enough of these interfaces in fiction to 
be able to learn lessons from them. 

Brain Interfaces: A Minefield of Myths 

It's certainly possible to imagine and depict new interfaces for new kinds of 
thought aids. In fact, the foundations of these are already in the productivity 
tools we use every day— from word processors and spreadsheets that help us 
outline and express our ideas and model complex numerical interactions, to 
specialized tools such as task lists to aid our memory, Prezi presentations 
for modeling hierarchical thought, concept-mapping software, flowcharting 
software like OmniGraffle, and scientific search tools such as Wolfram 
Alpha. There is a lot of space to explore new interfaces that rethink how we 
think and aid more complex representations and development. We just need 
to put a little more thought into it. 

Brain Interfaces 155 



What Counts? 




Sensor Display 


Location Awareness 


Context Awareness 


Goal Awareness 


What's Missing? 


Augmented Reality Will Make Us Laser-Focused, 

Walking Encyclopedias 



Star Wars Episode IV: A New Hope (1977). 

Luke Sky walker rushes out of the farmhouse to the top of a ridge. 
Looking through a pair of binoculars, he scans the horizon for his 
new droid, R2-D2. "That R2 unit has always been a problem," C-3P0 
complains. "These astro droids are getting quite out of hand. Even I can't 
understand their logic at times." The viewfinder shows data at the edges of 
Luke's view, but it doesn't help (Figure 8.1). "How could I be so stupid," he 
whines, "He's nowhere in sight. Blast it!" 

Augmented reality (AR) is technology that augments a user's perception 
of the real world with useful, additional information. Though any of the 
senses could be augmented, AR is almost always visual in nature. It appears 
in a number of technologies in our survey, including binoculars, weapons, 
communications systems, heads-up displays (HUDs) for pilots, and even 
inside cybernetic eyes. 

What Counts? 

As its name suggests, AR is about augmenting reality, not replacing it. 
Representation of reality doesn't count. In Chrysalis, for example, Dr. Briigen 
cannot compare the volumetric projection of her telesurgical patient with the 
real thing because the patient is thousands of miles away (Figure 8.2). 

This example illustrates one of the constraints of AR. Because it is tied to 
reality, a user cannot readily manipulate the representation for scale, position, 
or state. It is conceivable that a system could pass between an AR mode and a 
manipulable representational mode, but our survey doesn't reveal one. 

158 Chapter 8 

Chrysalis (2007). 

Additionally, to augment reality, the information should overlay reality. 
The holoimager from Firefly floats its radiographic image above the patient 
(Figure 8.3a). To check it against reality, the user has to glance downward, 
so this speculative interface is just outside of our definition of AR. The Lost 
in Space volumetric display, in contrast, overlays Judy on the surgical table 
(Figure 8.3b). This allows Dr. Smith to check her for outward signs and gives 
him a radiographic view of her internal organs at the same time and in the 
same place. For this reason the Lost in Space example is just within our 
definition, even though it is quite similar to the Firefly example. 

Virtual reality doesn't count either because, by definition, it replaces the 
user's perceived reality. It's conceivable that a virtual reality could be easily 
augmented for its users, but there isn't any such example in the survey. 

These criteria define the boundaries of AR, which must refer to reality and 
overlay it. What we have found that meets these criteria falls into four 
categories of augmentation: sensor display, location awareness, context 
awareness, and goal awareness. But before we describe these categories, 
a word about the appearance of AR systems. 

FIGURE 8.3a,b 

Firefly, "Ariel" (Episode 9, 2002); Lost in Space (1998). 

Augmented Reality 


Iron Man (2008). 


Augmenting reality without obscuring it is the primary challenge for 
any AR system. Nearly every example in sci-fi solves this problem with 
translucent overlays that allow most of the "real" view to show through. 
This allows cinemagenic camera shots that show the user and the interface 
simultaneously, which looks appropriately futuristic because such displays 
are not common today (Figure 8.4). 

Many AR interfaces use intricacy of shape and complex motion to 
communicate technological sophistication to the audience. More recent 
examples add information that is modal, appearing automatically when 
needed though rarely completely disappearing. The combination of these 
aesthetic choices would likely be problematic in the real world because of 
the high degree of visual strain and distraction. Until real-world exposure 
informs audiences differently, however, we suspect this trend will continue 
because it suits storytelling needs. 

Sensor Display 

The simplest examples of AR are those that display basic sensor information 
at the periphery of a display. Often this is "dummy data" written in obscure 
symbols or a fake alphabet to add credibility and the appearance of 
technological sophistication without needing to worry about actual meaning. 

160 Chapter 8 

FIGURE 8.5a,b 

The Incredibles (2004); Transformers (2007). 

Star Wars Episode IV: A New Hope included simple sensor information with 
its binoculars (see Figure 8.1). The magnifications are enhanced with what is 
probably something like bearing, distance, and magnification measurements. 

Though the audience can't always decipher the information, the implication 
is that it is useful for the user. In The Incredibles, the Omnidroid, a robot the 
villain constructs to take over the world, has an AR with lots of numbers and 
graphics displayed, but little sense can be made of it (Figure 8.5a). A similar 
AR from Transformers is even more inscrutable because it's based on an 
alien language (Figure 8.5b). 

In Iron Man, Tony Stark's HUD shows him a great deal of sensor data, 
including suit performance data, flight controls, and environmental 
information. It's worth mentioning that the HUD is not a model for 
legibility — the amount of information is too great, with too much 
distracting motion (Figure 8.6). 

Iron Man (2008). 

Augmented Reality 



As these sensor display interfaces illustrate, AR potentially 
obscures too much of the user's view. To avoid getting in the 
user's way, designers should use visual design that maximizes 
transparency while maintaining readability. Objects should 
be placed at the edge of the user's view when they are not 
needed, and adjacent to the locus of attention when they are. 
Because human brains are hardwired to detect motion, data 
pushed to the periphery risks triggering false positives in the 
user's peripheral vision. Therefore, movement at the edges of 
the display should be kept to an absolute minimum. 


One of the challenges with AR and HUDs is the focal depth dif- 
ference between the augmenting information and the real-world 
view being augmented. Users who must change focus con- 
stantly and quickly between the two— even when overlain— get 
eyestrain quickly. Imagine the difficultly Tony Stark might have 
refocusing between the glass in front of him and objects in his 
flight path. Ideally AR systems should have stereoscopic capa- 
bilities so the augmentations seem to be at the same distance 
as the thing being augmented. This way the user can access the 
information with a simple glance without having to fully refocus. 

One of our favorite, surprising, and subtle examples of sensor display in 
AR appears in an apologetic for the gunner seats aboard the Millennium 
Falcon in Star Wars. We shared this example in Chapter 6, but in short, this 
interface augments the silent dogfight in space with dramatic audio to 
increase the gunner's field awareness (Figure 8.7). It bears another mention 
here because it reminds us that augmentations can happen across more 
than the visual channel. 


Star Wars Episode IV: 

A New Hope (1977). 

162 Chapter 8 


When working with one sensory channel, consider alternative 
channels for the augmentation. These other channels might be 
better suited to the information and have the additional benefit 
of not obscuring the primary one. 

Location Awareness 

Another category of AR has the system displaying geographic information 
about the user's location. An early and unsophisticated example appears in 
Robocop. As Robocop searches for his past, he heads toward the location of 
James Murphy's old house. En route, his HUD shows the names of streets as 
he approaches. When he is on the right street, the interface switches to a more 
precise mode that displays the address for which he's looking (Figure 8.8). 

A more recent and sophisticated example appears in Minority Report. 
Evanna is the field officer in charge of the search operation for the fugitive 
John Anderton. She remains in the pilot seat of a dropship to coordinate the 
operation while other officers enter a building to conduct a search. The upper 
right portion of her HUD shows the team's location on the city map along with 
several layers of additional information, such as the compass bearing of the 
ship. This same map appears in the lower half of the view in full perspective, 
adding schematics to the real view through the HUD (Figure 8.9). 

^ c g -4* 

m K/ uinf OSQ : f M 

j \ f r 


SV, I - 


: '_~\-- 

** I 

FIGURE 8.8a,b 
Robocop (1987). 

Augmented Reality 



Minority Report (2002). 

A particularly cinemagenic type of location awareness is terrain modeling. 
In both Aliens and Iron Man, characters navigate difficult terrain in dim 
conditions. In response, their interface augments the terrain with bright 
cyan contour lines, helping to make the location more navigable and giving 
the audience a strong sensation of motion in the process (Figure 8.10). 

Neither example explains how the contour lines are known to the system. It 
could be a real-time presentation of radar or sonar data, in which case it's a 
sophisticated type of sensor display. Especially in the case of the Iron Man 
suit, though, it seems more likely that the system accesses a topographic 
database correlated to the suit's current GPS and altimetry data. A third, 
more remote possibility is that it functions more like a human, recognizing 
the terrain visually and drawing on top of what it sees. This would make it a 
sophisticated example of context awareness, discussed next. 

FIGURE 8.10a-c 
Aliens (1986); Aliens 
(1986); Iron Man 


Chapter 8 

Context Awareness 

Context awareness includes the system's awareness of objects or people in 
the environment. 

Object Awareness 

Some AR systems display information about objects and people in view of 
the user. The type of information shown depends both on the system and the 
object or person, and sometimes its context. In the following example, Tony 
Stark's HUD has identified that the children he can see in magnified view are 
riding the Santa Monica Ferris wheel (Figure 8.11). 

Recall that in this scene, Tony is in the act of flying his experimental, 
supersonic, and weaponized suit for the first time. Tempting him to read 
an encyclopedic entry about the Ferris wheel seems like an irresponsible 
distraction. Although his onboard artificial intelligence named JARVIS might 
be able to handle the piloting while Tony is engrossed in the minutiae of 
the 1893 World's Columbian Exposition in Chicago, a better solution, given 
JARVIS's presence, would be to provide just the name and schematic, and let 
Tony ask for more information. JARVIS could tell him what he wanted to know 
while letting Tony look around and monitor the flight visually. 


It's possible to augment reality with all sorts of rich, layered 
information, but the user only has one locus of attention, and 
their focus is on the task at hand. To avoid distracting them, 
limit content to what is either vital or explicitly asked for, rel- 
egating cues about access to further information to a distant 
second level of attention. 

FIGURE 8.11 
Iron Man (2008). 

Augmented Reality 165 

FIGURE 8.12 

Firefly, "Serenity" (Episode 1, 2002). 

In the pilot episode of Firefly, an Alliance cruiser spots the eponymous 
spaceship while on patrol and suspects it of illegal salvage. While the 
Alliance commanding officer discusses what to do with the hapless heroes, 
the bridge viewport in the background encircles the view of the ship with 
additional information (Figure 8.12). 

The tricky bit about this scene is the augmentation graphics. They are 
properly aligned for the camera and officers. But what do the soldiers on the 
left side of the bridge see? The parallax would be wrong for them, putting the 
augmentation far to the right of the ship. In fact, all overlays must take into 
account the viewer's perspective. Shared HUDs like this one can't do what 
it appears to do. Apologizing for episode writer and director Joss Whedon, 
perhaps this cruiser has a super-advanced display technology that beams 
separate overlays to each viewer's eyes and then crops the overlay to the 
boundaries of the viewport to reinforce that the object being augmented is 
outside the ship. But that's pure apologetics (see p. 268 for a similar parallax 
issue with medical volumetric projections). 


It's possible that augmentations can be coordinated between 
AR displays such that each viewer sees the same thing as if it 
were "in reality," but this should be kept to high-level or public 
information, such as street signs or general warnings. Doing so 
leaves room for the personally relevant sort of information that 


Chapter 8 

is the promise of AR systems, like directions to a place oth- 
ers might not be headed to. In the case of the Alliance bridge, 
for example, this would mean that different officers would see 
different augmentations. The communications officer might be 
able to see graphics describing whether the communications 
channels are jammed. The weapons officer could see whether 
the ship's shields were up. The science officer might get a read- 
out of any radiation or chemical emissions from the ship. 

Awareness of People 

People are social creatures, so AR sometimes augments social interaction. 
In the time-traveling adventure Back to the Future Part II, future Marty is 
speaking with his boss, Douglas Needles, over a large videophone in his 
living room. During the course of the conversation, the system shows some 
peripheral information about Needles, including his age and occupation 
(Figure 8.13). 

In Iron Man, Tony has access to similar biographical information on his HUD 
about his friend Rhodey while on a telephone call with him (Figure 8.14). In 
neither scene do characters make use of this information, so it is difficult 
to assess if it was the right stuff to display. It could be based on the users' 
preferences. We suspect that future systems could use the conversation itself 
as a source for determining what information is relevant. 

FIGURE 8.13a,b 

Back to the Future Part II (1989). 

FIGURE 8.14a,b 
Iron Man (2008). 

Augmented Reality 


FIGURE 8.15 

Terminator 2: Judgment Day (1991). 

A useful bit of augmentation in combat is to know who in view is friend 
or foe. In Terminator 2: Judgment Day, the T2 cyborg has AR built into his 
vision. When assessing the threat from soldiers in the Cyberdyne lobby, the 
overlay tells him to "select all targets," which we know he has done after he 
incapacitates them all in a matter of seconds (Figure 8.15). 


We do not see a circumstance in this film where there is a mix 
of friend and foe in one view, so we don't know how the T2 
AR would display it. We know the system has the capability of 
highlighting individual objects, so it could visually distinguish 
the dangerous ones. This view, then, must be a special mode 
for dealing with cases in which everyone is a threat. Rather 
than presenting a lot of overlays and requiring the T2 to spe- 
cifically interpret the noisy signal, checking to see if any one 
of them needs to be singled out for protection or incapacita- 
tion, the special mode lets him know at a glance that he can 
be indiscriminate in his action. Designers should similarly help 
their users quickly recognize cases in which either everything 
or nothing is to be considered. 


Chapter 8 


— " 

- — ri 

4- ■ 

- : 

m ■■■.ii-mi j .1111114 ■■; -■;■''' ' '!! '" " ! ' 


£V Hi 

Lr -%i-- ^J . 

r!HHr ' ^Bi 

I II 1w A ' & 

FIGURE 8.16a-c 
Iron Man (2008). 

Even with fast threat assessment, the T2 still has to manually incapacitate 
the targets. Other, more recent friend-or-foe systems perform the 
identification, targeting, and trigger pulling. The Iron Man HUD shows its 
work in progress as it assesses who in view is a hostage and who is an enemy 
(Figure 8.16). Though the suit fires multiple mini rockets automatically after 
the assessment, it overlays reticles above each target, coloring the foes red 
before it fires its missiles and kills them. 

The suit can probably do this friend-or-foe assessment much, much faster 
than its display shows. Why this artificial slowness? 


The results of computer processes have consequences. The 
direr those consequences, the more human oversight is needed 
to safeguard against false negatives and false positives. Pro- 
vide a first checkpoint by slowing processes down to the speed 
at which a human can decide whether intervention and correc- 
tion are necessary. 

Augmented Reality 


FIGURE 8.17 
District 9 (2009). 

In District 9, the semi-autonomous alien exosuit HUD displays blue 
corner reticles when identifying individuals, and then color codes 
entities' silhouettes based on a DNA match (Figure 8.17). Human foes 
are automatically attacked. Though this scene occurs while the suit is in 
automatic mode, we can presume that when the main character Wikus is 
occupying it, he sees the same display and can act accordingly. 


Humans see faster than they read, and when time is critical, 
design cues must be seen and understood quickly. Iron Man's 
reticles well communicate what's about to happen, but they are 
more visually complex than the overlays of color in District 9. 
Iron Man's reticle aids targeting, but if the computer system is 
doing the targeting, a simpler mechanism, such as the one in 
District 9, is called for. 



The friend-or-foe examples have the luxury of augmenting 
people in a constructed world where those in sight are either 
one or the other. But the real world is likely to be more compli- 
cated. Certainly, there are clear threats, such as a man aiming 
a loaded rifle at your head, and clear nonthreats, such as a 
sleeping infant in his mother's arms. But what about the loaded 
weapon not aimed at you? How does the system handle these 
in-between states in a way that both alerts the user to the nu- 
ances but is quickly understood for immediate action? 


Chapter 8 

FIGURE 8.18 
Robocop (1987). 

Goal Awareness 

The most advanced AR systems combine some type of identification (sensor, 
location, object) within the context of the user's goals. Knowing these goals 
helps systems prioritize what is to be shown, and when and how it is to be 
displayed. These goals can be very broad, such as Robocop's prime directives 
(Figure 8.18). 

Alternatively goals can be more concrete, such as flying or targeting. 

Goal: Flying Well 

A common goal-aware AR involves flying. These systems take important 
elements from the instrument panel and display them on a HUD for the pilot. 
Most of these, such as altimeter and airspeed indicators, might be described 
as sensor displays, but the false horizon is one that is optimized for the goal 
of keeping the craft level. Each of the examples below have targeting AR as 
well (Figure 8.19). (For more on targeting, see the next section.) 

In Iron Man, we get to see the HUD of a US Air Force pilot and compare it 
with one in the Iron Man suit. The Iron Man HUD presents more data with 
more intricate graphics, and has object recognition and a full-screen display. 
This interface is remarkable for its modality as well, bringing information to 
the screen intelligently as it is needed (Figure 8.20). 

Augmented Reality 171 

— H 

r i i i 


-— IBH^^^ta 


4 J 4 /- /." 

FIGURE 8.19a-c 

Aliens (1986); Independence Day 

(1996); X-Me/7 (2000). 

FIGURE 8.20 
/z-o/7 Ma/7 (2008). 

Goal: Precise Targeting 

The most common AR interface found in the survey is used for targeting. In 
these examples, animated reticles in a targeting viewfinder help guide the 
shooter's eye, magnify the view to allow for more precision, or change state 
to let the shooter know when a hit is more likely (Figure 8.21). 

Reticle is the formal name of the set of crosshairs and lines that help a 
shooter aim precisely at a target. The design of reticles varies greatly in 
sci-fi properties. Aside from focusing the user's attention on a point in the 
middle, they vary in color, shape, intricacy, and motion. (And brand! If you 
look closely, you'll see that Mai's AR from Fireflybears the Weyland-Yutani 
logo from the Alien franchise. There's some interface evidence that Alien is 
in the same diegesis as Blade Runner, too, implying a terrifying universe of 
xenomorphs, Replicants, and Reavers.) 


Chapter 8 

Mi/ \ ^ 


FIGURE 8.21a-d 

Predator (1987); Robocop (1987); Starship Troopers (1997); Firefly, "Serenity" 

(Episode 1,2002). 

The assassin Zam Wesell's weapon interface in Star Wars Episode II: Attack 
of the Clones combines fixed and dynamic reticles nicely, with the brackets 
from the fixed reticle repeated in fading layers as the target moves. This 
creates an arrow-like tunnel that points in the direction that the weapon 
must be aligned, and provides a logarithmic emphasis to the moment when 
the weapon should be fired (Figure 8.22). The augmentation doesn't just alert 
her to information; it helps her achieve her goals. 

FIGURE 8.22 

Star Wars Episode II: Attack of the Clones (2002). 

Augmented Reality 




Only the Iron Man targeting system is smart enough to place 
the reticle, distinguish friend from foe, and pull the trigger. 
Computer systems are surely much more efficient than humans 
at this task. Why don't we see it more often? We hope that this 
rarity is because sci-fi makers aren't comfortable putting an 
algorithm in charge of life-or-death decisions. Still, it would be 
useful for users in an unambiguous situation to pass control to 
the software and then take back control when more difficult 
decisions must be made. 



Much of the time and effort of firing a weapon comes from 
manually aiming the weapon at something the eye is already 
targeting precisely. If the AR system could monitor the shooter's 
gaze— something already possible with today's eye-tracking sys- 
tems—then the weapon could be aimed automatically and the 
system would be much more efficient. This would free the hands 
to do other work, such as confirming the target is an enemy. 

An impressive example of goal awareness appears in Terminator 2: Judgment 
Day. After time traveling to the present day, the T2 gets its bearings and 
walks toward a pool hall. Along the way, the AR shows simple compass 
information and notes in passing that the nearby vehicles are relevant to its 
goal of finding transportation (Figure 8.23a, b). This confirms that the AR 
system is tied to the T2's goals in a seamless way. 

Later in the movie, young John Connor insists that the T2 not kill people. The 
T2 agrees as long as this limitation doesn't interfere with its goals, and the 
AR displays feedback about adherence to this requirement (Figure 8.23c). 

Human goals are often like this, remaining strategically the same while 
tactics are adjusted to suit the situation, new information, and constraints. 
A useful goal-aware AR should accommodate this. 

174 Chapter 8 



FIGURE 8.23a-c 

Terminator 2: Judgment Day (1991). 

Augmented Reality 175 

What's Missing? 

Despite all of the examples in this chapter, our survey doesn't provide an 
instance of one particular aspect of AR: interaction with the AR systems. 
Tony Stark interacts with JARVIS vocally, but in so doing bypasses most of 
the input challenges by relying on the even more advanced technology of 
artificial intelligence. Other ARs are sophisticated enough that they show 
relevant information at just the right moment. We never see an example 
of a user needing to change the augmentation to something more useful 
when the system gets it wrong, or dismissing an augmentation that is in the 
way. The challenges of interacting with an AR to change the display can be 
significant if the user is occupied in some engrossing task. 

Augmented Reality Will Make Us 
Laser-Focused, Walking Encyclopedias 

AR is a relative latecomer to the sci-fi canon of technologies, and because it 
is not commonly available commercially, still feels futuristic. These displays 
are cinemagenic and exciting, and sci-fi makers keep adapting available 
technology into them — HUDs, GPS, encyclopedic online data, and real- 
time image processing — giving rise to much of what the examples show. As 
sci-fi struggles with the more difficult and forward-looking features of goal 
awareness and interactivity, we can anticipate it setting the bar for real- 
world AR when it becomes widely available. 

176 Chapter 8 



Humanness Is Transferable to Nonhuman Systems 179 

Appearance 185 

Voice 186 

Audible Expressiveness 188 

Behavior 189 
Anthropomorphism: A Powerful Effect That 

Should Be Invoked Carefully 195 

Sam Bell is the solitary human worker overseeing a mining base on 
the moon. His only companion, GERTY, is an oversized robotic arm 
attached to the ceiling of the living quarters. When he talks to GERTY, 
it answers him in both spoken and iconographic forms. 

Recently, Sam has begun to suspect there's something strange going on. 
"Hey, GERTY, since I've been up here, I've sent Tess over a hundred video 
messages. Where did those messages go? Did they ever reach her?" 

GERTY answers, "Sam, I can only account for what occurs on the base." 
Though its calm male voice doesn't convey any emotion, its emoticon display 
switches between uncertainty and unease (Figure 9.1). 

Sam asks, "What about the messages she sent to me?" 

GERTY repeats, "Sam, I can only account for what occurs on the base," but 
its emoticon now changes into a smiley face (Figure 9.2a). 

When Sam finally decides his fears must be true, he asks, "GERTY, am I really 
a clone?" The emoticon display switches to a blank (neutral) face (Figure 9.2b). 
The answer is likely to prove unpleasant to Sam, and the system is calculating 
how best to respond. When GERTY finally decides to tell Sam the truth, it 
displays a sad face, as if empathizing with Sam and his newfound discovery 
(Figure 9.3a). 

When GERTY tells Sam that he's a clone and explains why he has false 
memories, Sam silently turns away. GERTY responds by crying (Figure 9.3b). 


Chapter 9 

FIGURE 9.3a,b 
Moon (2009). 

GERTY isn't human and doesn't possess any human attributes other than 
a synthesized voice. It doesn't look human and the emotions it expresses 
are little more than flashcards. Yet, GERTY does seem to sympathize, 
understand Sam's experience, and respond much like a human companion 
would. GERTY is a machine, but also a sympathetic character. Why? 

Humanness Is Transferable 
to Nonhuman Systems 

This is a common phenomenon in both sci-fi and interface design. Most sci-fi 
films and TV shows in the survey include examples of anthropomorphized 

In the example above, Sam knows GERTY isn't human. So do we, the 
audience. Yet Sam treats GERTY as a companion, not merely a set of 
subroutines. Likewise, R2-D2 is one of the most beloved characters in the 
Star Wars films, yet this little droid neither looks nor sounds human. In the 
real world, we speak to our cars as if to coax them into making it to the gas 
station before they run out of gas, and we curse our computers when they 
behave unexpectedly or when their programming doesn't make sense to us. 
Like Sam, we know these systems aren't alive and don't understand us. Yet 
we treat them as if they were a living being. 

The first thing to understand about anthropomorphism is that people can 
and do anthropomorphize almost everything— from hurricanes, teddy 
bears, and pets, to furniture, tools, and machines. It seems we have evolved 
specific mental equipment to understand other humans that we bring to 
bear in our relationships with most everything around us. 

Anthropomorphism is a fundamental psychological bias about which many 
books have been written. For our purposes, we only want to look at the ways in 
which this principle applies to technology. In their work at Stanford University, 
Clifford Nass and Byron Reeves have shown in controlled experiments that 
people, whether they realize it or not, tend to deeply anthropomorphize any 
sufficiently sophisticated technology— whether a car, a microwave oven, 
or even a company. Their research, supported by B.J. Fogg at Stanford, has 



shown that people give computer systems a full range of social considerations, 
ascribe to them human motivations, assign demographic attributes to them 
such as age and gender, and react to them in social ways, such as through 
persuasion and flattery, all the while being unaware that they are doing so. 1 
The successful systems, these researchers explain, conform to human social 
norms. Designers and engineers aren't responsible for anthropomorphizing 
their systems— users do that themselves. Instead, they are responsible for 
developing the systems so that they become acceptable characters, instead 
of annoying ones, by having them conform to social rules. Though a full 
examination of those norms and related social cognitive biases are beyond the 
scope of this book, it follows that designers who want to exploit this effect in 
the interfaces they design should look into them. 2 

Some technologies are designed specifically to trigger this anthropomorphic 
sense. The ASIMO robot, for example, is designed to appear and move quite 
like a human (Figure 9.4a). For less humanoid systems, people respond 
idiosyncratically. For example, some people name their Roomba vacuums 
and speak of them almost like a pet but others do not (Figure 9.4b). 

FIGURE 9.4a,b 

Honda's ASIMO robot (c. 2000); iRobot's Roomba vacuum robot (c. 2002). 

Reeves, B., & Nass, C. (1996). The media equation: How people treat computers, television, and 
new media like real people and places. New York: Cambridge University Press. 

As a plus, you'll be able to charm cocktail acquaintances with terms such as "outgroup 
homogeneity bias" and the damning "Dunning-Kruger effect." Begin your search with the 
phrase "social biases." 


Chapter 9 

Human Isn't the Only Possibility 

Astute readers may note that in sci-fi, technology doesn't have to mimic humans. It can 
mimic alien species as well, and even animals and plants. These have subtly different 
triggers and effects worth investigating. 

For example, the robotic daggits from the original Battlestar Galactica series (Figure 9.5a) 
and the Bounty Bear search program from Until the End of the World (Figure 9.5b) seem 
clunky and fake, whereas the teddy bear from A.I. seems wondrous, like it could become 
a good friend. The difference is the fidelity of the representation. In addition, the use of 
animals as the representation, instead of people, conveniently lowers expectations for 
users toward that of a pet or companion instead of an equal. 

FIGURE 9.5a,b 

Battlestar Galactica (1978); Until the End of the World (1991). 


Because most systems can't come close to mimicking the hu- 
man behavior necessary to interact appropriately in a social 
context, animals, plants, and aliens can be used instead. This can 
create an emotional connection in users without raising expecta- 
tions above the capabilities of the system. The result is often an 
endearing character or technology instead of an annoying one. 

In the case of Until the End of the World's Bounty Bear (see 
Figure 9.5b), only the crude animated representation and 
voice distinguish it from today's Google search capabilities. 
Yet the sum total has an endearing quality one would never 
claim for Google. 



FIGURE 9.6a-c 

Minority Report (2002); Metropolis (1927); Sfar IVars Episode IV: 

A New Hope (1977). 

So what aspects of a system might provide those triggers? Humans are 
complex and have many qualities that a device could emulate. Our review 
of anthropomorphic examples in the survey tells us that they fit into the 
following broad categories: appearance, voice, and behavior. These categories 
are not mutually exclusive. Gort, the robot from The Day the Earth Stood Still, 
for instance, emulates human anatomy and behavior, but it has a silver visor 
where a face should be (see Figure 6.1). The Minority Report" spy ders" display 
the intention of finding and "eyedentiscanning" citizens as well as problem- 
solving skills, but physically they don't look at all human (Figure 9.6a). 

From the first robot in Fritz Lang's 1927 film Metropolis to the very 
familiar C-3P0 and R2-D2 in Star Wars, robots are a prime example of this 
phenomenon (Figure 9.6b, c). 

In some cases, the robot is indistinguishable from a real human, like 
Ash, the "artificial person" in Alien (Figure 9.7a). They can also be mostly 
humanlike but with telling differences, such as Star Trek: The Next 
Generations Lieutenant Data (Figure 9.7b); somewhere in the middle, such as 
the "3 Laws Safe" robots in 7, Robot (Figure 9.7c); or only vaguely humanoid, 
like Robby the Robot in Forbidden Planet (Figure 9.7d). 


Chapter 9 

FIGURE 9.7a-e 

Alien (1979); Star Trek: Generations 
(1994); /, Robot (2004); Forbidden 
Planet (1956); The Terminator (1984). 

Even when the appearance is convincingly humanoid, this doesn't mean 
that the behavior will be. The T-800 terminator in the original Terminator 
film looked real enough but was monotone and robotic in speech and 
movement (Figure 9.7e). Likewise, the wax figures in Madame Tussaud's 
maybe convincing in appearance, but they fail in every other aspect of being 
humanlike. Depictions that are almost but not quite perfect feel creepy to 
most people because we have evolved to be deeply sensitive to humans that 
are "off," indicating that they may be either sick or ill intentioned. Behaviors 
and appearances that trigger this discomfort are said to lie in the uncanny 
valley, a term coined by researcher Masahiro Mori to describe "the revulsion 
many people often feel for human facsimiles" (Figure 9.8). 

Once a system takes human form or adopts humanlike behavior, it 
implies that the system has humanlike capabilities. It also triggers 
social conventions we normally reserve for other people. Our shorthand 
mechanisms for dealing with each other kick in and we start cajoling our 
cars, naming our computers, and treating that shopping agent as if it really 
knows what it's doing. These effects become more apparent when we look at 
the categories of things that trigger them in technology. 




Human likeness (or behavior) 


The uncanny valley. 


People are comfortable with human facsimiles until the similar- 
ity becomes so strong that it stops registering as a thing made 
to look human and becomes a human that has something 
wrong with it. Up until the valley, we accept all sorts of repre- 
sentations, but only because we set our expectations accord- 
ingly. The problem comes when the representation outpaces 
the system's functionality as well as its ability to conform to 
proper behavior. This is a very real effect and it holds for all 
types of representation— voice, image, gesture, proportion, 
and so on. It's critical for designers to understand this effect as 
their creations approach the valley: venture too close and they 
become repulsive, too far and they don't register as human. 


Chapter 9 


To avoid overpromising capabilities to your user, provide clear 
signals of the non-humanness of your systems. For highly 
humanoid-appearing technologies, alter some feature that your 
user interacts with the most, such as the eyes. If your interface 
speaks to the user, have the system speak with a carefully 
stilted vocabulary. Finally, if your system displays anthropomor- 
phic behavior, consider making it just stiff and robotic enough. 
These signs set user expectations at a lower level, avoiding the 
uncanny valley. 


The first and most apparent aspect of humanness is in appearance. It can 
mean just the body, just the face, or just the eyes. It can vary from vaguely 
human to indistinguishable from human. In the Matrix films, programs 
are represented in a virtual reality as fully human characters, imparting 
greater impact, depth, and danger to the audience than almost any other 
representation could. The hunt-and-destroy program, called Agent Smith, 
feels more dangerous and capable than one would expect from a program 
(Figure 9.9a). The prediction program, called the Oracle, seems wiser and 
more trustworthy when represented as a cookie-baking matron than lines of 
code (Figure 9.9b). 

Agent Smith and the Oracle are programs represented as lifelike characters 
who think, react, show initiative, and emote on occasion right along with 
the actual humans in the Matrix, triggering characters and audiences 
alike to ascribe human motivations, intentions, and constraints to them, to 
their detriment. 

FIGURE 9.9a,b 
The Matrix (1999). 




Many interfaces in sci-fi feel human because they sound human. This could 
be through the sense of "having a voice" through the use of language. It can 
also mean audible expressiveness without formal language. 

We need to be careful not to overgeneralize though. Signs, books, and websites 
use language, but they aren't anthropomorphic. It is the interactive give-and- 
take of conversation that signals a humanlike, responsive intelligence. 

This responsive use of language can be in text with no accompanying voice, 
like we see in the artificial intelligence called Mother in the movie Alien (see 
Figure 3.3). It answers questions and displays a crude sense of intention. 

More frequently this use of language is embodied in sci-fi by an actual 
voice, which greatly increases the sense of humanness, beyond just the 
language used. In the case of the TV series Knight Rider, K.I.T.T. is the 
onboard artificial intelligence assistant in the car. Almost the entirety of its 
development as a character on the series is accomplished through K.I.T.T.'s 
voice. There is a minimal text interface, used rarely, a voice-box light that 
glows with his speech, a scanning red light at the front of the car, and 
K.I.T.T sometimes controls the car itself, but this is the extent of its behavior 
(Figure 9.10). The bulk of our acceptance of him as an independent character 
is due to his very humanlike voice, which includes intonation, sophisticated 
phrasing, and natural cadence, timbre, and annunciation, unlike more 
robotic-sounding voices. 

FIGURE 9.10a,b 
Knight Rider (1982). 


Chapter 9 


Because the effects of anthropomorphism aren't consistent for 
all users, some may interpret the use of language as a part of 
the system while others may interpret it as the system itself. 
For example, one driver of a car with a voice recording might 
associate the voice with a part of the car (perhaps the safety 
system), while another might associate the voice with the 
whole car (as with K.I.T.T.). Still another might think that the 
voice is an actual person speaking through a telephone-like 
connection. Each of these agents would have different capa- 
bilities, and mistaking the wrong set of capabilities could be 
frustrating. Make it clear to users what the voice represents. 

In 2001: A Space Odyssey, the HAL-9000 computer has a human-sounding 
voice, including intonation, though the timbre and cadence is much smoother 
and less expressive — even soothingly so. It's part of the reason why HAL's 
willingness to sacrifice the crew late in the film seems so sinister. The voice 
implies personality and humanness where they don't exist, so what seems 
like a calm, logical manner when everything is fine turns menacing and 
psychopathic once HAL considers the crew a threat to the mission. In both 
cases, HAL is still the same set of emotionless instructions. 

A missile launched from a ship — even a self-guided one — maybe "smart" but it 
isn't considered sentient and certainly isn't a character. But if it reveals a voice, 
it suddenly seems to have a mind of its own — and quickly becomes a character. 
In the cult film Dark Star, a member of the crew tries to convince a warhead 
equipped with an artificial intelligence that its countdown was triggered 
erroneously. The conversation ends on an existential note — right before the 
bomb explodes (Figure 9.11). Similarly, in the "Dreadnought" episode otStar 
Trek: Voyager, B'elanna Torres tries to convince a missile she reprogrammed to 
abort its mission with a similarly deep, existential argument. To her credit she 
succeeds where the crew oiDark Star failed. In both cases the bombs' voices 
and use of language trigger anthropomorphic senses. 

FIGURE 9.11 
Dark Star (1974). 

Anthropomorphism 187 

If a computerized voice sounds more mechanical than human, it will be 
understood to be an artificial system; however, when a machine system has a 
natural human vocal representation, it can cause unexpected confusion. For 
example, when the Atlanta airport train (now called the Plane Train) opened 
in 1980, the cars had no human operators on board, but were equipped 
with a prerecorded human voice to give instructions and announce stops. 
What the designers didn't foresee is that some riders presumed the voice 
was coming from a human conductor able to make judgment calls when 
managing the trains. Riders would take more chances, such as rushing 
doors as they were closing, because they expected the conductor to see them 
and wait. The realistic voice created unrealistic expectations of how the 
system would behave. To solve the problem, computerized voice recordings 
(sounding just like the Cylons in the original Battlestar Galactica TV show) 
were substituted. These were not as "natural" or "comfortable" for riders, 
but they set the proper expectations. (This topic has been a hot one for more 
than 20 years. One of the authors was at a contentious panel discussion 
about this very subject at a conference way back in 1992!) 

Audible Expressiveness 

All of the above examples involve language, but emotive sound can also 
express an anthropomorphic sentience, too. In the Star Wars films, R2-D2, 
one of the most endearing characters in the franchise, does not speak a 
human language, and he certainly doesn't look at all human, either. Still, his 
beeps, chirps, and whirrs are emotive enough for audiences to understand 
when it's feeling fear, excitement, and disappointment. These emotions are 
part of what tell us it's a sentient, anthropomorphic character. 


The sound effects you choose to communicate with users 
have the potential to create a character of the system. To be 
effective, these sounds must have an evocative character in 
how they respond to system events. For example, a sad sound 
accompanying an "Error 404— Page Not Found" might express 
sympathy for the user's frustration. This can make a system 
more endearing but, like other uses of anthropomorphism, can 
raise expectations among users that extend past the system's 
actual capabilities. 

188 Chapter 9 


At its heart, anthropomorphism is behavior centric. People see faces in most 
everything, so appearance is an easy win, but humanlike behavior increases 
the sense of anthropomorphism greatly. With humanlike behavior, even the 
most mechanical things seem to gain personhood. One well-known example 
is from Pixar's short film Luxojr., in which two lamps, through movement 
alone, tell an endearing story of the exuberance of childhood. There's 
nothing humanoid about these objects, but their movements suggest a head, 
face, and hips, and their relationship to each other convince us that this pair 
is deeply human (Figure 9.12). 

Another example from sci-fi is in Iron Man, as Tony Stark's robotic helper — 
affectionately called Dummy— becomes a believable and endearing 
character simply through lifelike movements while responding to Stark's 
conversation, despite its utilitarian, industrial robot appearance and lack of 
sound or voice (Figure 9.13). 

FIGURE 9.13 
Iron Man (2008). 




If the technology for which you are designing an interface has 
the capability to move, consider consciously designing that 
motion to make users more comfortable and communicate 
capabilities or system status. Though it would require particular 
responsiveness, users have a built-in capability to understand 
and empathize with things that do so. 

Anthropomorphism can even occur with behaviors limited to text responses 
and button clicks. For example, back in 1966, a computer program named 
Eliza created a stir (and controversy) by imitating a Rogerian psychologist 
who simply asked questions based on previous answers. It had a very simple 
algorithm and a pool of starter questions to ask but exhibited remarkable 
flexibility in "conversing" with people. In fact, some users were completely 
deceived into believing they were dealing with a highly sophisticated 
psychoanalysis program, and, even more remarkably, others reported 
personal insights despite knowing that the program was just a few lines of 
trick code. Even when we know the system is a simulation, it is still possible 
to build a successfully anthropomorphized experience — if the purpose and 
constraints of the system are appropriately focused. 

Degrees of Agency: Autonomy and Assistance 

Another behavioral trigger for anthropomorphism in sci-fi is agency and 
autonomy. Agency in this context refers to a system's ability to carry out 
known actions per predefined parameters. Autonomy refers to a system's 
ability to decide to initiate new actions to help achieve a goal. 

Many examples of both exist throughout sci-fi. Most robots, like R2-D2 and 
C-3PO, have both agency and autonomy, as do systems like K.I.T.T. from 
Knight Rider. Characters in Star Trek's holodeck have agency and a limited 
autonomy, but when they gain full awareness and autonomy it usually spells 
big problems for the crew. This example illustrates why it's important to 
distinguish these because autonomy is both more powerful and riskier to 
associate with anthropomorphism. 

Consider Ebay's auction systems or stock-trading services that can keep 
placing trades up to a set amount, and even complete a purchase without 
needing our intervention. That's agency. It's a system we trust enough 
to spend money on our behalf, given its restrictions. Ebay's system isn't 
anthropomorphized, but if it were, it might have an impact on how people 
use it, and how often. 

Now consider if the system had autonomy. In this case, it wouldn't just act 
for us, it would decide for us: that we liked but lost a bid on that previous 
bookcase, and this other one is similar, so it will go ahead and place a bid 

190 Chapter 9 

on it for us. It might find, buy, or even sell items without our intervention. 
A stock management system might do the same, choosing to buy or sell 
stocks we haven't even considered, not merely the ones we already own. 
At the heart of any of these systems would be the necessity of trust. Which 
system is more trustworthy— the one that acts or looks anthropomorphic 
or the one that doesn't? The answer partly depends on the style and degree 
of verisimilitude, of course, but all things being equal, the research tends 
to support the idea that most people trust systems that exhibit human 
characteristics more than those that don't. 3,4 

Assistance is the least degree of agency for anthropomorphic systems. 
An agent designed for the purpose of assisting a user with answering 
questions or completing a task is called & guide. A guide might help you find 
something, but not find it on its own. It might make suggestions on how to 
write a letter, but it wouldn't write it for you. 

There are few examples of guides in sci-fi. One is Vox, the library interface 
in The Time Machine (2002). Vox is a virtual reference librarian projected 
into an upright pane of glass that helps time traveler Alexander Hartdegen 
use the 2030 New York Public Library. Stepping into a futuristic library, 
Hartdegen sees a row of vertically mounted glass panes bisecting the length 
of the hall (Figure 9.14a). As he approaches a bookshelf contained within a 
glass case, a translucent figure appears as if on one side of the glass behind 
him and introduces itself: "Welcome to VOX system. How may I help you?" 
After Dr. Hartdegen muses, "Oh, a stereopticon of some sort," the machine 
replies, "Oh, no sir. I am a third-generation fusion-powered photonic with 
verbal and visual capabilities connected to every database on the planet. A 
compendium of all human knowledge. Area of inquiry?" The visual display of 
the photonic can watch a user and even meet his or her gaze (Figure 9.14b). 

Enthralled, Dr. Hartdegen asks, "Do you know anything about physics?" 
The photonic replies, "Ah . . . accessing physics . . ." and raises its hand to 
the surface of the glass where a rectangular information browser labeled 
"Physics" appears and cycles through various diagrams. As Dr. Hartdegen 
asks for "mechanical engineering," "dimensional optics," "chronography," 
"temporal causality," and "temporal paradox," he is amazed to see the device 
instantly summon additional browsers for each of these topics. At first 
the photonic seems quite excited by his genuine scientific interest, but as 
Hartdegen continues requesting topics, Vox's expression turns perturbed 
and dismisses the browsers to clarify, "Time travel?! . . . accessing science 
fiction" (Figure 9.14c). 

3 Lee, J. L., Nass, C, & Brave, S. (2000). CHI '00: Extended abstracts on human factors in computing 
systems. New York: ACM. 

4 King, W. J., & Ohya, J. (n.d.). The representation of agents: Anthropomorphism, agency, and 
intelligence. Retrieved from 

Anthropomorphism 191 

FIGURE 9.14a-c 

The Time Machine (2002). 


Why would a learning interface convey attitudes— positive or 
negative— about requested topics? Such a negative attitude 
about a particular topic could serve the cultural purpose of en- 
couraging certain types of learning, such as practical science, 
and discouraging others, such as creative expression, while not 
outright prohibiting them. That this pressure is due to the fact 
that the interface looks human probably adds to its effect, but 
designers need to be careful about when, how, and if social 
cues should cast opinion on learning. 

The real world doesn't have enough artificial intelligence to create truly 
autonomous agents yet. They're strictly products of sci-fi at the moment. 
And there are only a few examples of good agents. We see plenty of examples 
of guides outside sci-fi, however, in real or prototyped products, including 
Apple's Knowledge Navigator and the Guides 3.0 prototypes, Microsoft Bob, 
Microsoft Office's Clippy, and myriad "wizard" interfaces. It's a step outside 
of sci-fi, but to get interactive anthropomorphism right, we need to head 
there for a bit. 


Chapter 9 

FIGURE 9.15a-c 
Microsoft Bob's characters 
and home screen (c. 1995); 
Microsoft Office's Clippy 
(c. 1997). 

Two famous examples from Microsoft were unsuccessful guides, 
underscoring how difficult it can be to do these well. Despite the fact that 
much of the research behind the personalities of Microsoft Bob, a personal 
information manager, and Clippy, an assistant inside Microsoft Office, was 
based on the work of Nass and Reeves (see page 179), the behaviors exhibited 
by these products were deemed nearly universally annoying by users (Figure 
9.15). A clear mismatch existed between expectations and actions in terms 
of behavior. In particular, Clippy was often interruptive and presumptuous 
about its help, overpromising and underdelivering. It was difficult to turn off 
permanently. Software often transgresses social norms, but Clippy felt worse 
because it seemed like a social being that should have known better. It had 
two eyes, it used conversational language in response to your behavior. It 
behaved like a real person — a really annoying person. 


This is particularly true in terms of agency, autonomy, authority, 
and cooperation. Because of this, all system behavior should 
conform to social rules appropriate to users' cultures in order 
to be fluid, helpful, and appropriate, regardless of any outward 
appearance or representation of humanness. This isn't easy and 
isn't easily generalizable. Designers must have firsthand knowl- 
edge of what is appropriate and what isn't. 



FIGURE 9.16a,b 
Apple's Knowledge 
Navigator (1987); 
Apple's Guides 3.0 

[SB Daily Life on the Oregon Trail- 1843 

Tht politJIcaL status of the Oregon. Country was still a subvert of 
disputebetween the United States and Britain in 13-43. but prospective 
settlers- nevertheless gathered in such, towns as Independence, 
Missouri, to begin the long tr«k westward. One of these ptoneers was 
Jesse App]egate. who published y^ears later his own account of the 
movement of the giant wagon train that eventually reached the 

The migrating body numbered o^er 1,000 souk with about 1 20 
wagons, drawn by six ox teams,, averaging about six yokes tc the team, 
and several thousand loose horses and cattle. 

The emigrants first organised and attempted to tra ve] in one body, but 
It was soon found that no progress could be made with a body so 
cumbrous. ar.das yet SO averse to all discipline. And at the crossing o: 
the * big Blue" it divided into two columns, which traveled In 


( S^Kbl ][ Ituitt j 


:■■..■••:: jj 'v.-::-.:.. "J 

It is possible to get guides and agents right, though. One real-world 
example comes from Apple. In Knowledge Navigator, an industrial film 
created in 1987 to show possible future technologies in action, Phil, a partly 
realistic animated agent, assists a college professor in a variety of tasks 
before his upcoming lecture (Figure 9.16a). Phil works because he is easily 
interruptible, doesn't presume too much knowledge, and isn't represented 
with too much realism. This helps signal that although he's advanced, 
he's not as capable as a human assistant would be. He conforms to social 
conventions appropriate to his capabilities. 

The other example was a working, experimental system of guides for 
learning. It was created in Apple's Advanced Technology Group to 
demonstrate new database technologies, though it never made it to market. 
Guides 3.0 had four guides, all with algorithms controlling their behavior, 
and all characterized as people — three as content guides in period costume, 


Chapter 9 

and one as a system guide named Brenda (Figure 9.16b). The content 
guides offered perspective and further material on entries in a historical 
database of the United States. They were specifically chosen to represent 
different— and differing— points of view on the content in the database, and 
their behavior was designed to reflect how much they had to add, or not, 
to any particular entry. The purpose was to make clear that history is open 
to interpretation and to offer just a few perspectives. In this way, teaching 
history was also teaching students to form, accept, and acknowledge 
their own point of view. It was as if Hartdegen from The Time Machine had 
three other guides available to him in addition to Vox that could help him 
interpret information from different perspectives and not just find it. 



People have different learning strengths and multiple ways of 
understanding. Offering users a single point of reference may 
give them confidence, but at the cost of omitting other useful 
points of view. Offering users multiple perspectives, either from 
the system or other users, can help people understand the com- 
plexity of the material or situation in a way that suits them best. 

This probably wouldn't be a good strategy for instructions in 
an emergency or information that isn't open to interpretation. 
When the information isn't definitive or can't be clearly applied, 
however, differing points of view offer users a mechanism for 
considering advice from experienced others without the impli- 
cation that there is one right way to proceed. 

Anthropomorphism: A Powerful Effect 
That Should Be Invoked Carefully 

People are used to interacting with other people, so this plays into our 
experiences of technology in both sci-fi and the real world. In sci-fi we see 
software, robots, cars, and search engines take on aspects of humanity to 
make it easier for the characters and easier for the audience to relate to and 
understand. Studying these examples shows that humanlike appearance 
makes people more comfortable and helps interfaces communicate more 
expressively. Humanlike behavior can make systems more instantly 
relatable and be well suited to assisting users with accomplishing tasks and 
achieving their goals. 

Designers wanting to incorporate anthropomorphism should take great 
care to get it right, however. Anthropomorphism can mislead users and 
create unattainable expectations. Elements of anthropomorphism aren't 
necessarily more efficient or necessarily easier to use. Social behavior may 

Anthropomorphism 195 

suit the way we think and feel, but such interfaces require more cognitive, 
social, and emotional overhead of their users. They're much, much harder to 
build, as well. Finally, designers are social creatures themselves and must 
take care to avoid introducing their own cultural bias into their creations. 
These warnings lead us to the main lesson of this chapter. 


When we design technologies to be anthropomorphic, it raises 
user expectations about the extent of their capabilities for intel- 
ligence, language, judgment, autonomy, and social norms. If your 
technology cannot fulfill these expectations reliably, you risk 
frustrating your users. Design signals into the appearance, lan- 
guage, and behavior that communicate clearly that the system is 
not human, so that expectations and reality are well matched. 

One way to hedge bets with the phenomenon of anthropomorphism is 
to ease into it with the half-step of zoomorphism. Because people have 
lower expectations of animals yet still find themselves building social and 
emotional connections with them — particularly their pets — representing 
a system as an animal rather than as a human can often score the positive 
attributes without risking a slip into the uncanny valley. This maybe the 
more effective strategy while we continue to learn more about developing 
systems whose behaviors can truly stand up to social conventions and 
possess what users might consider intelligent behavior. 

196 Chapter 9 



Asynchronous versus Synchronous Communication 199 

Specifying a Recipient 203 

Receiving a Call 208 

Audio 214 

Video 217 

Two More Functions 218 

Communication: How We'll Be Talking Next 221 

Metropolis (1927). 

' CrI 

J oh Frederson walks to a large wall-mounted device. While checking his 
messages on ticker tape (Figure 10.1a), he sees that Grot in the Lower 
City needs his attention. Reaching up to the dial on his right, he turns 
its hand counterclockwise from 10 to 6. Then he turns the left-hand dial 
to 4, and the screen comes to life. It displays "HM 2" overlaying a shifting 
blend of different camera images. Joh fiddles with a few controls to clear the 
signal (Figure 10.1b). 

Once he has Grot in view, Joh picks up a telephone handset from the device 
and reaches across to flip a button on and off to signal Grot. In response, 
the lightbulbs on Grot's videophone begin to flicker and make a sound. Grot 
rushes to his device, looks into the screen, and lifts his handset. His screen 
comes to life with Joh s image, and the two have a conversation (Figure 10.1c). 

Communication technology makes up the largest proportion of any 
type of technology in the Make It So survey. This is no surprise because 
communications technologies — including cinema itself— are among the 
most radical changes to our sense of space and time and to ourselves both as 
individuals and as cultures. There are so many examples of communication 


Chapter 10 

tech that there is no way to survey them all, much less reference them in this 
chapter, but even the ones in the survey provide clear enough patterns to 
structure the chapter. 

Asynchronous versus 
Synchronous Communication 

Communications technology can be distinguished in several ways. 
One helpful distinction is whether the communication is synchronous 
or asynchronous. Synchronous technologies have the communicators 
interacting in real time, both attending the communication simultaneously, 
as on a telephone call. Most of the communications examples in the survey 
are synchronous. Asynchronous communications involve a sender encoding 
his or her communication in some medium, such as a letter, video, or audio 
recording, and then sending it to the receiver. The terms asynchronous 
and synchronous are an eyeful, so we'll talk about the more friendly words 
message and call, respectively. These two types overlap quite a bit, but where 
they don't is largely a function of composing, editing, and sending messages. 


Composing a message requires that you record the message in advance. It 
also gives you time to reflect on what you've recorded and change it if you 
like. Recording a text message would require handwriting tools, a keyboard, 
or some transcription tools, but we don't see as many of these in action as we 
do the time-based media of audio and video messages. Some alien controls 
for composing messages are impossible to interpret, as when Klaatu 
prepares an audio report using gestural controls in The Day the Earth Stood 
Still (see Figure 5.1). 

Of the rest, audio messages are seen much less frequently than video 
messages. The controls for these interfaces are the basic ones that would 
appear on a digital video camera. A standard example appears in the movie 
Sunshine, as Robert Capa prepares a final video message for his family. 
On the screen we can see that he has controls to review, delete, send, and 
record his message (Figure 10.2a). The film doesn't show the corresponding 
physical controls. 

Other examples of such tools include Penny's video log from Lost in Space 
and Jake Sully's video log horn Avatar (Figure 10.2b, c). It's worth noticing 
that neither interface provides on-screen recording controls. It's as if the 
filmmakers believed that recording devices were already so familiar to 
audiences that controls needn't be shown. What is shown in the Avatar 
example are four simple buttons with a somewhat confusing information 
hierarchy: main cam, past entries, submit, and task. 

Communication 199 

FIGURE 10.2a-c 

Sunshine (2007); Lost in Space (1998); 

Avatar (2009). 

These interfaces share one important feature. They all have a prominent 
signal to indicate that the system is currently recording. Sunshine and Avatar 
show the familiar red dot indicator that flashes. Of all the possible cues in a 
recording device, this is the one they almost universally have in common. 


If the design of a recording interface is too busy, it can distract 
and annoy both the audience and the sender, who needs to 
focus on his or her message. Reducing the interface to only 
the minimum needed while recording helps the sender focus 
on the message, but there should still be some signal that the 
system is in fact recording. This signal doesn't just help the re- 
corder but in sci-fi it also helps the audience understand what 
a character is doing. The most common visual icon for this 
signal is a blinking red dot. This signal is so common that using 
another visual cue might confuse users and audiences alike. 

No other midrecording information is displayed in the interfaces seen in 
the survey, such as duration of recording, time remaining, or a sense of how 
much has been recorded so far. Similarly, no editing interfaces are seen with 
which a character could review and edit the message. 


Chapter 10 


A recipient needs a way to play recorded messages. In addition, sensitive 
messages may require the recipient to be identified, as seen in Star Wars and 
The Incredibles (Figure 10.3). In the former, R2-D2's artificial intelligence 
does the identification. In the latter, the message plays automatically upon 
identification of Mr. Incredible. In each case, the recipient doesn't have to 
initiate playback manually. 

More often, recipients have control of the playback. Commonly, only 
Play and Stop controls are seen. These controls follow the predominant 
technology paradigms of the time: toggle switches, buttons, or touch-screen 
controls. Little if any screen time is given to these controls, and usually only 
the Power and Play functions are used (Figure 10.4). 

FIGURE 10.3a,b 

Star Wars Episode IV: A New Hope (1977); The Incredibles (2004). 

FIGURE 10.4a-c 

2001: A Space Odyssey (1968); 

Brainstorm (1983); Starship Troopers 




Activating the System 

Communications technologies need to be able to be turned off for privacy, 
quietude, and power conservation. The separate On and Off functions are 
often combined into a single control and are almost always physical toggle 
switches, such as the flip-open cover of a Star Trek communicator (Figure 
10.5a). Some are innovative alternatives, such as the touch-activated 
combadges first seen in Star Trek: The Next Generation (Figure 10.5b). 


Users appreciate easy controls. The Star Trek communica- 
tors use large, easy motions to activate and deactivate. The 
combadges are positioned so that tapping them is easy. You 
can imagine that these systems could be activated in even 
easier ways, such as by simply touching the communicator. But 
when activation becomes too easy, it can become error prone. 
Designers must ensure that the controls are easy to use but 
protect against accidental activation. 


One advantage that the handheld communicator has over the 
combadge is that anyone can tell at a glance whether the com- 
munication channel is open or closed. Though audible signals 
indicate when the combadge is connecting or disconnecting, 
when someone walks into a room wearing one, its status is un- 
clear. It would be easy to use one to covertly record a conver- 
sation. A better system would clearly indicate its current status 
as well as when that status changes. This is true for almost all 
systems, not just communications. 

FIGURE 10.5a,b 

Star Trek: The Original Series (c. 1968); Star Trek: The Next Generation (c. 1987). 

202 Chapter 10 

Specifying a Recipient 

Whether for a message or a call, the sender needs to specify a recipient. 
This task can be accomplished in a variety of ways: automatically through a 
fixed connection, with the help of a system operator, by specifying a unique 
identifier such as the recipient's phone number, or by a stored attribute such 
as the recipient's name. 

Fixed Connection 

Some communications devices are tied to companion devices and 
are unable to communicate with any others. Some of these broadcast 
continually while activated, such as the headsets seen in Aliens, and the only 
interface needed is a switch to turn them on. 

If the technology is designed to broadcast to many people, the sender only 
needs to activate a switch to indicate that he or she is speaking. This can be 
a momentary switch on a microphone, or a hook on which the microphone 
rests, as seen in the ship address system used by Commander Adams in 
Forbidden Planet (Figure 10.6). 

FIGURE 10.6a,b 
Forbidden Planet (1956). 






FIGURE 10.7a,b 

Aliens (1986); Chrysalis (2007). 

Semipublic interfaces such as intercoms are fixed to a recipient, but the 
interface needs a way to request the receiver's attention (see "Notification," 
page 208). In Aliens, when corporate stooge Burke visits Ripley at her 
apartment, he presses a button on the intercom to get her attention 
(Figure 10.7a). Thereafter, the receiver can flip a switch to open continuous 
communications, or engage in push-to-talk exchanges as with walkie-talkies 
(see "Audio," page 214). These communications can smoothly flow into action, 
as happens in the film Chrysalis, as Hoffman lets Clara into his home. She 
rings, he glances at the video, and after making up his mind, he presses a lock 
icon on the intercom touch screen to let her in (Figure 10.7b). 


The trade-off between control and clarity is longstanding in 
interaction design. Giving an interface more controls means 
there is more to learn and distinguish, and too many controls 
can crowd the interface with unnecessary visual noise. Typi- 
cally, control works better for experts, and clarity works best 
for novices. As a rule of thumb, however, closely related func- 
tions, such as activation and connection, can be combined into 
a single, easy-to-use control. 


When a device can connect to any other on a network, callers need a way 
to connect to the right one. One solution is to have an operator— whether 
a human, alien, or artificial intelligence — make the connection. What is 
important is that the operator can understand the caller's request and 
communicate back when there is a need for clarification or to convey a 
problem with the system. Most often in sci-fi these requests are spoken, but 
there's no reason why these requests couldn't be some other input, such as 
written or gestural. 

204 Chapter 10 


New technologies can be complex or unfamiliar enough to re- 
quire user training, but this may be impractical or scale poorly. 
An alternative is to train certain people to act as operators 
on behalf of users. This strategy offers many benefits that an 
automated system might have difficulty with. Aside from lan- 
guage barriers, people need no training to communicate with 
operators. The operator can interpret the user's intentions and 
emotions, and respond accordingly. Operators can also handle 
problems that arise with the system— especially those that are 
unexpected by the system designers. The history of job spe- 
cialization can be considered an interface to the complexities 
of technology (operators), biology (doctors), law, politics, and 
even religion (clergy). Until we have perfected both android 
appearance and artificial intelligence, well-trained humans, 
despite their flaws, may remain the most usable interface. 

A Unique Identifier 

Despite their benefits, operators don't scale well to handle massive numbers 
of users. Letting communicators find one another across a system alleviates 
this burden, but requires that the system work for both the callers and the 
network. To solve this problem, many networks use a unique identifier 
(UID). This is the strategy of modern telephone systems that use telephone 
numbers as UIDs. The telephone system is the predominant first-person 
communication technology, and it is also the predominant paradigm in 
sci-fi. These interfaces almost always use a numeric keypad, whether analog 
or screen-based (Figure 10.8a, b). Activating the keypad buttons in sci-fi 
follows conventions of the time and includes push buttons, touch screen, 
and in Johnny Mnemonic, a remote control with a laser pointer to indicate a 
number and a button to select it (Figure 10.8c). Most of these systems are 

t ^Hp 

FIGURE 10.8a-c 

Blade Runner (1982); 2001: A Space 

Odyssey (1968); Johnny Mnemonic 




direct entry, but these familiar systems pose an interface problem, in that 
they increase the chance for user error. 


Telephone number entry systems like the TouchTone system 
seen in 2001: A Space Odyssey are direct entry, meaning each 
number is registered by the system as a button is pressed or a 
number dialed (see Figure 10.8b). The challenge of this interface 
is that mistakes can't be undone and must be retyped from the 
beginning. This effect gets worse with longer numbers. 

A better solution for this problem is seen in modern mobile 
phones and in the example from Johnny Mnemonic (see 
Figure 10.8c). In these systems, a user enters the number to 
the system explicity and then submits an Enter or a Call com- 
mand once the entire number has been composed. This gives 
the caller the opportunity to review and correct errors before 
committing to the call. To avoid errors when asking users to 
supply a set of inputs, let them compose first, then review and 
correct before committing. 

Stored Contacts 

One problem with the unique identifier (UID) strategy is that it requires 
people to remember lots of long number strings, and people aren't generally 
good at that. Characters in sci-fi almost never have difficulty remembering 
these numbers, but this is a major concern for users of real-world 
communications technologies. As a result, storage systems such as speed 
dial, voice dial, and contact lists reduce the burden on users' memories, 
letting them remember a single digit or name for frequently called numbers. 
The owner of a device often has the burden of inputting and managing these 
shortcuts, but an interesting counterexample appears in Aliens. 

When Burke meets with Ripley, he leaves a calling card with her. When 
she finally decides to contact him, rather than remembering his number 
or even his name, she simply slips the transparent card into a videophone, 
and the system automatically connects to him. Note that it's his Weyland- 
Yutani business card, but the system finds him on a videophone at home 
that is presumably as immobile as the one Ripley is using (Figure 10.9). This 
interaction speaks to a great deal of sophistication. 

206 Chapter 10 


As technology becomes increasingly networked and ubiqui- 
tous, a person's whereabouts can be more and more confident- 
ly assessed. Presuming that privacy and disambiguation issues 
can be managed, this provides an excellent opportunity to save 
a caller from having to manage UIDs to contact particular de- 
vices. Instead, the caller should be able to merely identify the 
recipient and let the system worry about where she is and what 
devices she has with her. Could it be in the future that when- 
ever a phone near you rings, you should answer it, because it 
will be for you? 



What if instead of UIDs, the system could find who we want 
with more ambiguous data? It's not too far-fetched. Vir- 
tual assistant software, such as Apple's Siri, is already do- 
ing something similar with available locational information. 
Siri interprets what you ask for and checks against available 
location-relevant information automatically in the hope that 
it relates to what you want. In the future, a user might ask her 
phone, "Contact Joe, or maybe it was Joseph, in the Chicago 
area with whom I met at that conference in 2005?" or "Who 
was that guy I met at Jennie's party who said he was a financial 
consultant?" Eventually, will a person just think of a recipient 
and the computer takes it from there? 



Receiving a Call 

Communication is more than sending a message or placing a call. It needs to 
be received. This section looks at things from the recipient's perspective. 


Once a caller or a message finds its way through the network it needs to get 
the recipient's attention. When we look at these examples, a note of caution 
is called for. If a sci-fi story bothers to include a message in the plot, it is 
important. This gives these signals a weight that isn't applicable to all types 
of messages or calls in the real world. 

While Logan watches the Carousel ceremony in Logans Run, he receives 
a text message from central control letting him know that they have been 
assigned to handle a runner who is nearby. Carousel is quite loud, but the 
audio is conveniently loud enough to get his attention (Figure 10.10). 


Users want to attend to urgent calls and messages. To gain a 
user's attention, an interface should immediately send a signal 
to a sense that does not depend on a visual cue, because the 
user's eyes may have a limited range of perception at that 
moment. Hearing is one sense optimized to provide full-field 
awareness, making audible signals a good candidate. Ideally, 
audio alerts should be context-sensitive in order to set a vol- 
ume for the sound above the ambient noise and select a type 
of sound that contrasts with the ambient noise so it is more 
likely to be heard. Touch signals such as vibration require physi- 
cal contact with the device, making it less likely to be detected. 
Sci-fi technologies may have access to other full-field senses 
such as orientation, pressure, temperature, and smell, but 
they cannot carry as nuanced and discernable a meaning as 
sound, and real-world designers rarely have access to actuators 
that affect these. 

FIGURE 10.10 
Logan's Run (1976). 

208 Chapter 10 



Of course, not everyone can hear, and there are times when 
even those who can may be in an environment that prevents 
them from hearing an alert. Sound coupled with a vibration, 
a visual alert, or something that stimulates other senses is a 
good idea if the alert is truly important. 

In the time-travel comedy Back to the Future Part II, the family is at the 
dinner table when Marlene receives a call by putting on a pair of goggles. 
Red LEDs flash the word phone on the outside of the goggles as they ring 
(Figure 10.11). Marty Junior's goggles are already on, and he announces to 
Marty Senior that the phone is for him and that it's his supervisor Needles, 
implying a caller ID system, which had only just been released commercially 
in the United States the year before the film was released. 

Marty Senior takes the call in the den on the large video screen there. As 
he approaches the screen, it displays a portion of the Renoir painting La 
Moulin de la Galette, and has the blinking legend incoming call along the 
bottom (Figure 10.12a). When he answers it, the Renoir shrinks to a corner 
of the screen, revealing the live videophone signal (Figure 10.12b). During 
the conversation, the Renoir disappears, and text appears near the bottom 
of the screen to provide information about the speaker. This text appears 
automatically, with no prompting from Marty Senior (Figure 10.12c). 

In Iron Man, Tony is in his workshop on the lower level of his sprawling home 
when he sends a voice message to his assistant, Pepper Potts. The system 
finds Pepper in the living room and plays the message as the ring tone to get 
her attention. Simultaneously a display near her shows Tony's portrait and 
an alert reading incoming link. This audio draws her attention from the 
Mad Money TV program playing on the main screen (Figure 10.13). There are 
several notable things about this interface, and we will return to it below. 

FIGURE 10.11 
Back to the Future 
Part II (1989). 

Communication 209 

FIGURE 10.12a-c 

Back to the Future Part II (1989). 

^^ J 

■ Needles. Doufeyu/j. * r* 



h 1 

- 1 

FIGURE 10.13 
/ro/7 Ma/7 (2008). 


Many visual signals can be missed if placed too far from the 
user's locus of attention. To ensure that signals are received, 
it is best for designers to place them where the user cannot 
help but find them. If the system has eye-tracking controls, the 
place they are looking can be known exactly. Alternatively, if 
an input device such as a mouse has been used recently, it or 
its on-screen manifestation (such as the cursor) is an excellent 
proxy. Otherwise, place signals along the critical waypoints of 
the interface where the user's attention is most likely to fall. 


Chapter 10 

What We Don't See 

The survey hasn't revealed a recipient declining or rerouting a call through a 
visual interface, though both are options for such systems in the real world. 
We see examples of people doing this with human operators, as when Picard 
responds to a verbal message about an incoming communication with "I'll 
take it in my ready room," but we don't see examples of these functions 
outside of a conversational interface. 


Once a recipient has been notified of a call or message, he or she needs to 
connect to the call or open the message to review it. If the technology is 
commonplace in the real world, sci-fi adheres to the paradigm: for example, 
nearly all telephone calls are received by picking up a receiver, and nearly 
all text messages to mobile devices are received by pressing a button on the 
device. If the communications technology is not common, the show must 
develop some new way to accept the message. A few of these examples follow. 

As we've seen, characters in Star Trek: The Next Generation touch their 
combadge to accept an incoming call, routed through the ship's computer. 

That the combadge doesn't show recording status is discussed above, but the 
action for accepting the call is simple and accessible. Combadges are worn 
on the left breast, and after being notified audibly, the recipient only needs to 
tap it once to accept. The gesture is simple to execute and not likely to be done 
accidentally. We don't see a circumstance in which a crew member has his or 
her hands full but needs to take a call, but given their universal translation 
capabilities, it's likely that a voice command to accept would be understood. 

To receive Tony's call in Iron Man, Pepper reaches out and taps the notification 
message on the tablet screen. This opens an audio link between them, presents 
a stored photograph of Tony, and adjusts the layout so that the Mad Money 
video feed she was watching takes up less of the screen (Figure 10.14). 

FIGURE 10.14 
Iron Man (2008). 




Social pressures require that accepting a call be quick, and a 
desire to control one's own privacy requires that it be discrete 
and unmistakable. Until gestural controls become more ubiq- 
uitous (and corresponding communications technology more 
fully embedded in environments), tapping satisfies the user's 
need for speed and deliberateness. 

Monitoring the Connection 

As with recording, callers need to know when they are connected. This can 
be a signal that indicates when the link is disconnected, or it can be an 
indicator that persists while the connection is active. 

In blended-media telephony systems, both strategies can be used. Iron 
Man shows an example of this, as Pepper can see on her JARVIS screen 
a label confirming the active audio connection to Tony (Figure 10.15). It 
additionally provides a visual confirmation of the disconnection as the call 
panel on the display disappears when the call is over, adjusting the display 
so that the Mad Money video fills the screen again. 


Video calls provide continual feedback, so disconnection is 
usually apparent: the caller disappears or the image freezes. 
Audio calls are another matter because people take turns 
when speaking, and silence on the other end could mean either 
listening or disconnection. Any audio signal that served to 
confirm the connection would necessarily add noise to the call, 
and so such systems provide an audio signal only when the 
caller becomes disconnected. 

FIGURE 10.15 
Iron Man (2008). 


Chapter 10 

Ending a Call 

Typically, both parties to a call can end it when they want. There are a number 
of ways this is seen in sci-fi. Of course, we see examples that mimic real-world 
systems, such as returning a handset to its cradle, as an older telephone 
paradigm would imply. In systems that use tokens to represent recipients 
or initiate calls, removing the token ends it. In Aliens, for example, Ripley 
yanks Burke's business card out of the videophone to quickly terminate 
her uncomfortable call with him (Figure 10.16). There are also a few radio 
paradigms in which the caller turns a stop-dial or flips a toggle to sever the 
connection. More recently, computer and television metaphors are becoming 
more apparent, with a user pressing or tapping a button to end the call. 

The survey doesn't reveal any voice commands for ending a call, either to 
an operator or a voice control system, but given that Tony's hands are fully 
occupied while flying the Iron Man suit (and he does not appear to have ocular 
controls), his call with Brody must have been ended this way (Figure 10.17). 

FIGURE 10.17a,b 
Iron Man (2008). 

Communication 213 




r" = 

FIGURE 10.18a-c 
Johnny Mnemonic (1995). 

A very unusual means of ending a call is seen in Johnny Mnemonic, as 
Takahashi ends a frustrating video call by swiping his hand horizontally 
through the air, like a backhanded slap. In response not only does his call 
end, but the display retracts into the surface of his desk (Figure 10.18). 


We have to presume that the system does not require a mas- 
sive, arm-swinging swipe to end each of Takahashi's calls. The 
system can handle both the perfunctory wave that ends an 
earnings announcement as well as the angry slap that silences 
the rebuking caller. When technology allows for a range of 
inputs, users can channel emotion into them, making the input 
more than a control but also a medium of expression. If the de- 
gree of emotion affects the degree of system response, all the 
better. This means that the technology fits the way we com- 
municate as people much more readily than the precision that 
best fits computers. 


Another way of looking at communications technologies is through the 
controls particular to their medium or channel. For example, during a call, 
how does a speaker control when they want audio privacy and when they 
want to be heard? 

Some sci-fi interfaces don't have such controls and are continuously sending 
signals. The only way to mute them is to cover the microphone or turn off the 
system. These are most often military or aerospace applications. 

More commonly, audio interfaces adopt a push-to-talk paradigm like a 
two-way radio or pager. In these interfaces, only when the speaker holds 
down a momentary switch is their voice sent. These buttons often appear 
at the bottom of the interface, within easy reach and controllable without 
obscuring other parts (Figure 10.19). 


Chapter 10 

FIGURE 10.19a-d 

Buck Rogers (1939); Space: 1999 (1975); Star Wars Episode IV: A New Hope 

(1977); Firefly, "Safe" (Episode 5, 2002). 

The reverse is a mute function. 

A few microphones operate by proximity. One comes from the campy Sean 
Connery film Zardoz. The Eternals communicate with a central artificial 
intelligence called the Tabernacle through clear rings worn on the left hand 
(Figure 10.20a). The ring senses when it is near the mouth, and only then 
sends the wearer's voice. The sleeve microphone in Minority Report operates 
the same way (Figure 10.20b). 

Though we imagine that callers using videophones might have a similar 
need for a mute button, the survey didn't reveal an instance of such controls. 

FIGURE 10.20a,b 

Zardoz (1974); Minority Report (2002). 




Sci-fi often adds a visual layer to audio-only calls, especially if the camera 
lingers on the communications technology for a length of time without 
showing the recipient's face. 


When a communications technology doesn't have a visual for 
the person on the other end of the line, sci-fi supplies one so 
audiences aren't staring at a still image while the audio plays. 
Some are basic like the level meter seen on Pepper's call with 
Tony (Figure 10.21a). Sometimes it can be artistic and quirky, 
like the wall of tiny shutters from Barbarella's shipboard com- 
puter Alphie (Figure 10.21b). But one thing remains consistent: 
when levels are louder, the system displays more light. Though 
the inverse relationship is possible, as is the connection of any 
of the other measureable qualities of sound and light, the sci-fi 
default is that more sound equals more light. 

FIGURE 10.21a,b 

Iron Man, detail (2008); Barbarella (1968). 


Chapter 10 

What We Don't See 

In addition to lacking physical mute controls (mentioned above), we don't 
see volume controls for interfaces in sci-fi. The sound level is always perfect 
for the situation: loud enough to be heard over any other sounds yet quiet 
enough to be comfortable. This is a necessity in storytelling, and audio 
engineers carefully balance many channels of audio to get the mix just right, 
but from the perspective of the characters, their communications systems 
are able to expertly do the same thing. 


There are some interface issues particular to video as well. One that 
appeared early is that of the invisible camera. Videophones need cameras 
to capture the image of the speaker, but they're almost always missing 
from views. This holds true for large, flat video screens, like we see on the 
bridges otStar Trek spaceships, as well as volumetric projections. For sure, 
we see cameras in sci-fi — mostly for surveillance — but we don't see them 
for communications systems like videophones. One possibility is that the 
camera is "hidden" inside the screen, focusing through or between the 
image. Another possibility is that the display is capable of seeing as well as 
recording, making the entire screen a recording device. 

The one exception to the rule is 2001: A Space Odyssey, where the lens is 
apparent not only in incidental technologies, but it is also central to the 
imagery of the film, as the terrifying, ubiquitous, unblinking interface to the 
inhuman HAL artificial intelligence (Figure 10.22). 

While the invisible camera seen elsewhere deftly avoids the gaze-matching 
problem (discussed in Chapter 4), it raises important questions about 
privacy: How do you know when you're being recorded if there isn't even a 
camera lens? 

FIGURE 10.22a,b 

2001: A Space Odyssey (1968). 




The cameras in sci-fi interfaces are missing because neither 
the makers nor the audience want to think about the interface. 
They're focused on the meanings implied in the conversation, 
and the emotions on the speakers' faces. In other words, they 
are notably not thinking about the location of the camera. 
When the interface recedes, people can focus on what they 
care about— the social interaction. 

What We Don't See 

In addition to the absent volume controls discussed above, we don't see any 
privacy controls other than those that terminate the communication. 

Two More Functions 

Besides the fundamental communications issues listed above, there are 
two functions in sci-fi communications technology that bear mentioning: 
language translation and disguise. 

Language Translation 

Given the multicultural (and multispecies) universes in sci-fi, some sci-fi 
makers have given a nod to addressing how it is that all of the characters 
are speaking the one language that the audience happens to understand. 
Natural language processing of a single language is one of the toughest 
problems in modern computation, much less translating across different 
languages that evolved for different species' speech organs, brains, and 
cultures, and at vastly different points in their cultural evolution. Credit 
goes to those TV shows and movies that give the issue at least a nod: the 
translation chip from The Last Starfighter, the universal translators of the 
Star Trek franchise, and the Babel fish from The Hitchhiker's Guide to the 
Galaxy (Figure 10.23). Sadly, these are often throwaway technologies that 
do little to illustrate the complexities of their use, other than to reinforce 
the lesson above ("Focus on the Person, Not the Medium"). When and if the 
real world finally tackles these problems, these sci-fi interfaces will seem as 
quaint as Joh's wall phone in Metropolis. 

One of the oddities we see in Star Trek is with the universal translation 
built into the computer's communications. The system offers seamless, 
instantaneous translation between any known languages by any number of 
speakers, facilitating effortless communications. It is sophisticated enough 
to quickly parse and decode new languages quickly. It works for ship-to- 
ship communications and is built into the small communicators worn on 
uniforms in the Star Trek: The Next Generation series and later. 

218 Chapter 10 

FIGURE 10.23a-c 

The Last Star fighter (1984); Star 

Trek IV: The Voyage Home (1986); 

The Hitchhiker's Guide to the Galaxy 


The oddity in the technology is that we also see each speaker speaking our 
language in their voice rather than in their own language with a dubbed 
translation or subtitling. It's likely, of course, that this is just a convenience 
for the television makers, but it offers an opportunity through apologetics. 
What if this was actually a feature of the ship-to-ship communication 
system? What if at the same time it's processing language, it processes 
the speech organs of a speaker and overlays subtle changes so that the 
mouth makes movements consistent with the translation? This couldn't 
work for the in-person universal translators without some seriously sneaky 
augmented reality technology (for which there is no evidence in the series), 
but for ship-to-ship translations, it offers a useful lesson. 


Where possible, a translation system that matches a text 
translation with a similar translation of the sonic and physical 
parts of speaking will reduce cognitive friction on the part of 
the listener. Attentive readers may notice that this is similar to 
the gaze-matching lesson on page 84. In the case of speech, 
though, much more than head and eye position would need 
to be altered. 

For voices, this includes timing, diction, intonation, accent, and 
of course, the content of what was being said. This is compli- 
cated by the grammar and word differences that can take more 
or less time to say in a different language. A synchronizing 
system might need to increase or decrease speed significantly 
to fit speech and meaning into the same amount of time, and 
with the same emphasis, as the original speaker. It would also 
include all of the sonic qualities of the speaker's voice, such as 
pitch, breathiness, and timbre. Then once you add anatomy it 



includes altering the mouth, eyes, and facial expression of the 
speaker, all blended together to make them seem like they are 
fluently speaking another language. 

As an ambassador might tell you, the act of learning another language 
favorably connotes an interest in the people who speak it. Those who have 
taken the time to do so will want that fact recognized. Additionally, unless the 
translation software is absolutely perfect, listeners will need to know when the 
fault of an off-seeming remark might lie with the system. For these reasons, 
systems doing such translations should provide an unobtrusive signal to 
distinguish when someone is being translated and when they aren't. 


Given that communications technologies are, by definition, mediated, there 
is an opportunity for disguise that is only rarely seen in the survey. Here we 
turn again to Takahashi's deceptive calling system in Johnny Mnemonic, 
in which the wicked businessman Takahashi uses a gestural interface to 
control the video avatar of a trusted friend of Johnny's. On his end of a call, 
Takahashi moves his hand like a puppet over a scanner, and the system 
uses the input to make the avatar speak its parts of a predefined script 
(Figure 10.24a). From Johnny's end, he sees the avatar's head move and 
speak on a video monitor (Figure 10.24b). As a gestural interface, we first 
introduced this example in Chapter 5, and it is noted above in this chapter 
for the emotional way in which the call is ended. Here we note it for its ability 
to grant its user a near-perfect disguise as a part of the communication. 

It is only near-perfect because the avatar speaks in a stiff and stilted manner, 
which raises some questions about the input method. If the computer needs 
a physical guide to move the avatar, wouldn't Takahashi's own face be a 
much more easy-to-manipulate and true-to-form input than his hand? This 
cinematic trick certainly helps the audience understand what's going on, but 
as an interface it only serves us by illustrating an opportunity. 

FIGURE 10.24a,b 
Johnny Mnemonic (1995). 

220 Chapter 10 



One challenge for videophone adoption is that people often 
don't feel they're in a socially presentable state— especially at 
home. As technology advances enough to create real-time ren- 
derings of people well out of the uncanny valley (see Chapter 
9 for more of this anthropomorphic principle), why not help 
out with a little real-time cosmetic alteration? Digitally fix that 
cowlick, render a clean and pressed shirt over that stained one, 
improve your muscle tone, or maybe smooth a few wrinkles. 
It's easy to see where vanity might quickly get the better of us 
with flawless avatars that drift too far from the real thing. But 
many sci-fi authors have suggested that this is the natural evo- 
lution of mediated identity, and it is a primary opportunity for 
us to experience our posthuman selves. Maybe verisimilitude 
isn't the right goal at all. 

Of course disguise can always be used, as this example illus- 
trates, for unscrupulous reasons— especially by identity thieves. 
This raises a corresponding opportunity for a counterpart sci-fi 
technology that verifies the identity of the caller. 


How We'll Be Talking Next 

Every major advance in communication technology has expanded the 
horizon of what we could experience across time and space: mark making, 
writing, the printing press, photography, radio, television, and the Internet. 
Today the capabilities are so vast and so commonplace it is difficult to 
appreciate. We can watch in real time as the sun rises in different parts 
of the world. We can hear the voices of the long dead, the alien moans of 
whales singing to each other in the depths, and the mysterious static of 
deep space. We can send and receive messages from people thousands of 
miles across the globe in a matter of seconds. We can become enraptured 
in the fantastically realized lives of people inhabiting worlds that only exist 
on screens. These advances in communication are happening faster and 
faster and don't look to stop anytime soon. Sci-fi will help us make sense of 
our vastly expanded senses by using these same technologies to envision 
possible futures, of whom we may be speaking to, and how we may be 
speaking to them, next. 

Communication 221 



Direct Download 225 

Psychomotor Practice 227 

Presentation Tools 232 

Reference Tools 236 

Machines to Think With 241 

Testing Interfaces 244 

Case Study: The Holodeck 247 

Learning: Aiming for the Holodeck 255 

FIGURE 11.1 

Star Trek: Voyager, 

"Once Upon a Time" 

(Season 5, Episode 5, 


Ship's cook Neelix enters the room where his young friend Naomi sits at 
a computer, diligently studying. She has been sad since her holodeck 
storybook friend Flotter, who is made of water, was "killed" by the 
Ogre of Fire in a holodeck simulation called "The Forest of Forever." He asks, 
"What are you working on?" 

She turns around to explain, "I'm researching the evaporation of water." 


"Well, I've been thinking. Water doesn't just disappear when it's heated. It 
turns into invisible gas. So, if we could get the forest to cool down enough, 
Flotter might reliquefy." 

Neelix pauses a moment before commenting, "Clever." He knows that Naomi 
has just learned the science that will save her friend (Figure 11.1). 

Learning is important in sci-fi narratives, just as it is in life. It contributes 
to what writers call the character arc, or the way a character changes over 
the course of the story. And when characters learn about interfaces or 
technology that is new to them, the audience learns about them, too, without 
additional exposition. Because of this, it's no surprise that we find learning 
interfaces in the survey. They can be divided into six categories: 

1. Direct download of knowledge into the brain 

2. Psychomotor practice interfaces that facilitate practice of a physical skill 

3. Presentation tools used by teachers during lectures 

4. Reference tools for simple information queries 

5. Machines to think with, which assist learners in processing their 
thoughts and developing cognitive skills 

6. Testing interfaces that measure knowledge and intelligence 
224 Chapter 11 

Direct Download 

This first category is often used as a "cheat" to save time by showing 
characters learning something important to their development or to 
advance the story quickly. Outside of sci-fi, filmmakers often use a montage 
to show a lengthy learning process in a short time. In sci-fi, however, it's 
possible to simply invent a technology that makes this knowledge transfer 
happen in a matter of seconds. Sometimes an author wants a character to 
just know something that wasn't known before, in order to get to the crowd- 
pleasing action. The interfaces for this type of direct-to-brain knowledge 
transfer vary widely, as no real-world analogue exists. Seldom is the 
technology for such learning explained. 

The first direct download interface in the survey appears in Star Trek: The 
Original Series. In the episode "Spock's Brain," an alien race sneaks on 
board the Enterprise and surgically removes Spock's brain from his skull. 
On a distant planet, a search party finds the brain installed in a computer 
controlling the civilization's infrastructure. The citizens there are far too 
unsophisticated to perform such advanced surgery, which is even beyond 
that of the Enterprises Dr. McCoy. 

They find a learning device called The Teacher that transfers sophisticated, 
ancient knowledge quickly, but the process is painful and the knowledge is 
only retained for a few hours. McCoy uses the device to learn how to replace 
Spock's brain. To absorb the information, he stands within a transparent 
dome with rods radiating outward from it (Figure 11.2a). There is no 
indication how McCoy selects the lesson he wants to learn, but perhaps he 
simply thinks it. After some fun sound effects and suffering what appears to 
be a splitting headache, a wild-eyed McCoy understands what he needs to do 
and sets to work on the intricate brain surgery (Figure 11.2b). Even though 
the knowledge lasts only for a short time, this is long enough to allow him to 
surgically reinsert Spock's brain and reconnect all of the nerves (Figure 11.2c). 

FIGURE 11.2a-c 

Star Trek: The Original Series, "Spock's Brain" (Season 3, Episode 1, 1968). 

Learning 225 

We have deliberately eschewed hand-drawn animation in the survey, but in 
the animated feature Fantastic Planet, a major component of the film's plot is 
an interesting device that transmits narrated video clips directly to the brain 
of the wearer. These "lessons" are, in the words of the narrator, "engraved on 
their memories forever." Though the content resembles reference material 
(see "Reference Tools," page 236), the technology is a direct-download device 
because it imprints the information in the brain passively and permanently, 
albeit at the inconveniently poky pace of language. 

To use the device, an adolescent of the giant alien Traag race named Tivva 
places the thing, which is shaped like a thin horseshoe with small spheres 
along its length, over her head like a hairband. The device blinks and bleeps 
a number of times, and the "infos" appear in her mind's eye. To show the 
audience what is being learned, the wearer's forehead is overlaid with 
narrated videos of the lesson. There is no indication that she requests any 
of the topics, so it is likely that it is a long, sequentially structured series of 
lessons (Figure 11.3). 

The device is also used by the smaller, human Oms, who are able to gather 
together in a group within the band and "watch" the infos simultaneously. 
The band works through a wireless or empathic technology that requires it 
to be close to the learner's brain. 

The primitive Oms use their newfound information to vastly improve their 
culture and advance their technology, to the extent that they are eventually 
able to create rockets to escape from the planet and the oppression of the 
Traag (Figure 11.4). 

FIGURE 11.3a-c 
Fantastic Planet (1973). 

FIGURE 11.4a,b 
Fantastic Planet (1973). 


Chapter 11 

nT s?l 



FIGURE 11.5a,b 

Perhaps the most well-known knowledge-download interface appears in 
the film The Matrix. After Neo is liberated from the Matrix, he undergoes 
training in a small and self-contained virtual reality called the Construct. 
Knowledge of every martial art known is uploaded directly to his brain 
by a jack in the back of his head. The pilot named Tank begins the upload 
by means of specialized software, operated by a touch-screen interface 
and keyboard. As the upload progresses, a portion of Tank's screen shows 
illustrative figures labeled with the martial arts knowledge currently being 
uploaded, a wide progress bar along the bottom, and a spinning 3D brain 
that "fills up" with an opaque solid (Figure 11.5). 

After the upload, inside the Construct, Neo looks up at Morpheus and says 
with amazement, "I know kung fu." Morpheus replies, "Show me." They fight. 
(See more direct-download interfaces and the erroneous myths they expose 
about brain interfaces in Chapter 7.) 

Psychomotor Practice 

Some systems provide an interface for practicing a physical skill. Called 
psychomotor learning in learning theory, it involves coordination between 
the brain and body. 

An example occurs in Dune, when the young prince, Paul Atreides, 
undergoes combat training with a "fighter," and practices his ability to 
control the strange weapon known as a "weirding module." (Learn more 
about the sonic interface of this unusual weapon in Chapter 6.) The fighter is 
a mechanical device that descends from a hole in the ceiling above a practice 
space. The fighter has several rings stacked along its height, each of which 
has a weapon of some sort, such as blades, holes that shoot metal darts, 
scissor-like bayonets, and extending spears with razor sharp points. The 
fighter can spin each weapon ring independently to create an intimidating, 
whirling machine of death (Figure 11.6). 

Paul's valet, Thufir Hawat, summons the fighter with a spoken command, 
and just before the fighting begins, gives it a verbal command, "Make the 
range two meters." Thereafter, the fighter obeys some internal program to 
conduct combat. Paul is an expert with the weirding module, defeating the 
fighter's weapons one by one. 



FIGURE 11.6a-c 
Dune (1984). 

In Star Wars Episode IV: A New Hope, Luke Sky walker spends some of the 
travel time between Tatooine and Alderaan practicing his lightsaber skills 
under the watchful eye of Obi-Wan Kenobi. He stands facing a baseball-size 
sphere that floats at head height. The sphere has half a dozen small circles 
on its surface that can deliver a harmless but painful laser blast at Luke. 
The sphere hovers in the air, spinning, bobbing, and weaving, and shoots 
intermittently. Luke tries to intercept each blast with his lightsaber, and 
mostly succeeds (Figure 11.7). 

When Ben swoons at the moment the destruction of Alderaan causes a large 
disturbance in the Force, Luke disengages his saber. Apparently context 
aware, the sphere stops moving about and firing. It is only when Obi-Wan 
gathers his wits about him and moves Luke to his next lesson — defending 
himself without being able to see — does the sphere take up its sparring 
duties again. No other explicit interaction occurs to signal the device to 
pause or resume its firing. 

FIGURE 11.7 

Star Wars Episode IV: A New Hope (1977). 


Chapter 11 


People enjoy working with good teachers— particularly in social 
contexts. When designing learning interfaces, even for tech- 
nologies largely expected to be used by one learner at a time, 
let the system accommodate the participation of a teacher to 
observe, model, comment, and challenge the learner as he or 
she progresses. 

In the film The Last Starfighter, the protagonist, Alex, spends many of his 
summer hours playing the video game Starfighter (Figure 11.8a-c). He 
plays it because he enjoys it, but after beating the final level, he learns that 
it is actually a secret training and testing device that has been deployed 
throughout the universe to find the best candidates for recruitment into 
the Rylan Star League. His skills in playing the game are directly applicable 
to becoming an ace gunner in the spaceship Gunstar, defending the forces 
of good against the onslaught of the evil Kodan Armada. The controls and 
heads-up display of the Starfighter game are, of course, very similar to those 
of the Gunstar (Figure 11.8d-f). In this case, the interface is the bridge 
between the safe challenge of learning in the game and the deadly serious 
situation in real life. 

FIGURE 11.8a-f 

The Last Starfighter (1984). 




If the learning interface is meant to develop real-world skills, the 
closer it matches the real thing, the easier learners will find the 
transition. Though by definition the learning interface isn't the 
real thing, it can help build skills that are directly applicable. 


Though a game model doesn't work for every type of learn- 
ing, many of the tenets of good game design can turn learning 
from a tedious task into something fun and deeply engaging: 
with progressive challenges, integrative skill building, role 
playing, and so on. (If you want to read more on this topic, 
try Chapter 12 of John Ferrara's Playful Design, which focuses 
specifically on games for learning.) 1 

The military cadets in Starship Troopers break into teams to fight in mock 
combat similar to the real-world game laser tag. The weapons they use in this 
scenario fire lasers that are harmless in and of themselves, but when they strike 
sensors on a harness worn by an "enemy," the harness delivers a nasty shock to 
the wearer, disabling him or her for the remainder of the exercise (Figure 11.9). 

FIGURE 11.9a,b 
Starship Troopers 

.M #1 

1 Ferrara, J. (2012). Playful design: Creating game experiences in everyday interfaces. 
Brooklyn, NY: Rosenfeld Media. 


Chapter 11 


As the troopers in Starship Troopers are mastering basic skills, 
such as how to operate their weapon, distracting them with 
worry about being horribly shocked would be counterpro- 
ductive. Only after they master these component skills do 
the shocks need to be added. By letting the stakes gradually 
increase along with the level of mastery, systems keep learners 
in the engaging balanced state between boredom and being 

The one noncombat psychomotor interface found in the survey is the tennis 
trainer seen in Total Recall. At home, Lori turns on a device with a click, and 
a life-size, volumetric tennis coach appears in her living room. It repeats the 
same serve over and over (Figure 11.10a). 

As the display continues, a disembodied female voice repeats, ". . . and pivot 
. . . and serve . . . and shift . . . and stroke . . ." Lori stands behind the coach, 
watching it and mimicking its motion (Figure 11.10b). After a handful 
of these repetitions, the virtual coach blinks red twice as a tone sounds 
(Figure 11.10c). Then the voice congratulates Lori: "Very good. Perfect form!" 
The system is not only projecting the coach but scanning Lori's movements 
and comparing them to an ideal. 

These psychomotor practice interfaces teach through experience, replicating 
some aspect of using the skill in the real world, whether by a virtual model of 
ideal form, a safe space to fight in, or a stand-in for a real opponent. 

— i n i nflK 

FIGURE ll.lOa-c 
Total Recall (1990). 



Presentation Tools 

When it is not practical to bring an object to learners or the learners 
to an object to be studied, teachers can use technology to model it in a 
presentation instead. Most scenes of this type of learning in early sci-fi show 
individuals standing at lecterns and speaking to their audience, with no 
accompanying display of any kind. Even the prescient film 2001: A Space 
Odyssey (1968) did not augment its presentations visually. 

One exception appears in the original Star Trek TV series, when Spock briefs 
Captain Kirk about their new mission, using images on a monitor mounted 
near the ceiling (Figure 11.11). 

As this capability was becoming possible in the real world in the mid- to late 
1970s, presentations in sci-fi began to use accompanying motion graphics. 
This helped visually tell the story, as well as telegraph the importance and 
logic of an upcoming action sequence. The survey's first example of this 
narrative strategy is in Star Wars Episode IV: A New Hope, when General 
Dodonna presents the attack plan on the Death Star (Figure 11.12). Over 
the next decade, these presentations become more elaborate and three- 
dimensional (Figure 11.13). 


When a task is being presented, learners benefit from seeing it 
modeled visually over time. This helps them quickly understand 
how it should progress, what they must do, and how the sys- 
tem responds. In interaction, this can mean temporarily taking 
control of the interface and showing users what they should se- 
lect or how they should gesture before they are asked to do it. 

FIGURE 11.11 

Star Trek: The Original 

Series, "What Are 

Little Girls Made Of?" 

(Season 1, Episode 7, 



Chapter 11 

FIGURE 11.12a-c 

Star Wars Episode IV: A New Hope (1977). 



FIGURE 11.13a-c 

Star Wars VI: Return of the Jedi (1983). 



By the late 1990s, filmmakers could afford to show 3D presentations in 
more mundane educational settings, such as the biology lecture in Starship 
Troopers, which features a rotating volumetric display of a "bug" (Figure 11.14). 

In Star Wars Episode II: Attack of the Clones, Obi-Wan seeks help from Yoda 
to find the location of a planet missing from the Jedi archives. Yoda is in 
the Jedi training school, leading a class of young children in lightsaber 
practice. To illustrate his problem, Obi-Wan places a small sphere onto 
the tip of a thin, upright rod (Figure 11.15a). It immediately begins to 
glow, and a slowly moving volumetric display of many star systems 
appears (Figure 11.15b). Obi-Wan is able to point to the place where the 
planet should be (Figure 11.15c). After one of the young students solves 
the mystery, Obi-Wan ends the display by telekinetically summoning 
the sphere back to his hand. Though this is not exactly a lesson, the 
technology's presence in the classroom indicates that it is ordinarily 
used for teaching purposes, with different spheres containing different 
materials to display. 

FIGURE 11.14 
Starship Troopers 

\ I'V 


FIGURE 11.15a-c 

Star Wars Episode II: Attack of the 

Clones (2002). 


Chapter 11 

FIGURE 11.16a-c 
Serenity (2005). 

Part of the beautiful introduction to the movie Serenity is a school scene from 
River's childhood. (Though how much of it is constructed or hallucinatory 
is ambiguous.) The lesson is about the evacuation of "old Earth" and is 
illustrated by an animation of those events, which the instructor is narrating 
(Figure 11.16). The animation occurs on a wall-size video display encircling 
the classroom. After the presentation is complete and the instructor wishes to 
talk with the students, the display fades from view, exposing greenery beyond. 
There does not appear to be any interface or controls by which the instructor 
is changing the display. It maybe that she is narrating a large, silent video that 
is programmed to fade from view at its close. 



The use of voice, video, animation, text, and other media in a 
blended, seamless whole helps keep the learner's attention. 
It also affords the content designers the opportunity to put 
content in the medium that fits it best. If the interface helps a 
teacher to plan and give the presentation, this assists both the 
teacher and the learner. 



In a classroom environment, an engaging presentation is often the first 
step in getting students interested in a problem or new material. Highly 
cinematic displays of large and detailed motion graphics serve several 
purposes. These satisfy the need of the learners to learn the material, the 
need of the author to explain the material to the audience, and how the 
learner encounters it. In addition, these presentations are often visually 
and auditorily stimulating for both audience and learner, and a chance to 
showcase new or speculative technologies. 

Reference Tools 

The reference materials seen in sci-fi are also predictably cinemagenic. 
When a character needs to look up a fact or ask a question, the answer is 
often animated and/or narrated. 

The first reference interface in the survey appears in the movie Things to 
Come. As John Cabell, a leader of a technocratic city of engineers, shares 
with his granddaughter some history of their great civilization, he shows 
her clips of silent video that help illustrate his stories. The only controls for 
this device are a pair of dials to the left of the screen (Figure 11.17). How this 
simple dial provides access to the potential library of video is unclear. 

Another notable reference appears in Dune. Paul Atreides uses a handheld 
tablet device as part of his studies. The device rests on a display stand while 
on a desk, but can also work comfortably on his lap (Figure 11.18). The content 
of a small oval screen at the top is controlled by four large push buttons 
below. Paul presses buttons to summon narrated videos of requested topics. 
The encyclopedic content includes star maps, surveys of indigenous plants, 
information about social castes, the behavior of the giant worms on the Spice 
planet Arrakis, and the industrial processes of Spice mining. 

FIGURE 11.17 

Things to Come (1936, 

colorized version). 

236 Chapter 11 

FIGURE 11.18 
Dune (1984). 

One bit of historical interest from this scene is the information architecture 
of the device, which is needlessly tied to the space efficiency needs of a 
printed book. That is, when Paul requests the topic "Weather," he is shown 
a screen with white text reading Weather [see Storms]. Paul navigates to 
the cross-referenced topic "Storms" and begins to watch. Modern audiences 
familiar with Wikipedia recognize that there is no reason why a digital 
medium can't just automatically reroute from the requested keyword to the 
actual content, with a bit of text to reinforce the correct keyword. 


When working in a new medium, try to avoid merely replicating 
the interface from an old medium. Specifically when moving 
something from print to interactive media, try to remove the 
unnecessary work required of a passive medium like paper that 
can be done on the reader's behalf. 

Superman and Superman Returns adhere to the reference technology from 
the Superman comic books: Superman has, in his crystalline Fortress of 
Solitude, a platform bearing a bank of crystals (Figure 11.19a). Inserting 
a crystal into a slot activates a database of Kryptonian knowledge 
(Figure 11.19b). A recorded image of his father, Kal-El, introduces the 
database by telling him, "Embedded in the crystals before you is the total 
accumulation of all literature and scientific fact from dozens of other worlds 
spanning the twenty-eight known galaxies" (Figure 11.19c). 

To request a topic, Superman speaks a command or question to the crystalline 
walls, and Kal-El appears and narrates a volumetric projection display 
(Figure 11.19d). The interface appears to be a fully functioning artificial 
intelligence, but the dialogue later reveals it is just a Kryptonian automated 
voice response system built on top of a vast database that anticipates 
thousands or perhaps millions of queries and responses. (Readers may enjoy 
comparing this to the reference technology from The Time Machine, already 
discussed in Chapter 9 for its anthropomorphic qualities.) 

Learning 237 

FIGURE 11.19a-d 
Superman (1978). 

A screen-based version of a similar interface appears in the film The Fifth 
Element. The character Leeloo, who has been resurrected from a bit of alien 
DNA, must learn all she can about the human race she is meant to save. Her 
tool to do this is a screen-based reference that goes unnamed in the film. 
This tool is available in many places: in Cornelius's home, on the Flogiston 
Paradise transport, and aboard Zorg's spaceship (Figure 11.20). 

To use it, Leeloo either selects from an on-screen menu of alphabetized 
topics or calls up a topic directly by typing it on a keyboard. Within each 
topic, she is shown pictures and video. For the topic of war, she is shown a 
series of rapid-fire images that illustrate the concept. This sequence might 
be a film editor's exaggeration of Leeloo's rapid learning, but the audience is 
given no evidence that this is the case. 

FIGURE 11.20a-d 

The Fifth Element (1997). 


Chapter 11 

These two interfaces illustrate one trade-off between command-line and 
WIMP interfaces (each described for their graphic qualities in Chapter 3). 
The Krypton database is very easy to use: Superman just asks his question 
aloud. Some of his questions aren't in the system, though, and it's a process 
of trial and error to learn what those missing questions are. He could waste 
a lot of time trying to figure out a way to re-ask questions only to ascertain 
that it's not how he's asking the question, but that the answer just isn't there. 
In contrast, Leeloo's interface looks less easy to use. She has to read the long 
menu of options, understand them, and make a selection. Presuming she's 
a competent typist, the costs she bears in parsing the interface pay off in 
helping prevent her from wasting time requesting an entry that isn't there. 
Each of these interfaces illustrates one side of a balancing act: Superman's 
shows ease of use; Leeloo's shows error prevention. 


Requiring users to select from a list of options avoids input 
errors and sets their expectations of what is available, but for 
vast amounts of content, it can be cumbersome. Giving users 
free-form inputs may be easier for them to express themselves, 
but it introduces problems of resolving unexpected input, 
disambiguating search terms, and hit-or-miss strategies of 
guessing what's available. When dealing with large amounts of 
content or options, designers should strike a balance in their 
search and navigation, setting expectations of what's gener- 
ally available, providing free-form input, suggesting most-likely 
content from context, helping them understand when they're 
about to request something that isn't available, and pointing 
them to alternate content or destinations when they request 
missing content anyway. 

Another note of caution about Leeloo's interface: the topics aren't clustered 
by meaning or by their connection to one another, but alphabetically, which 
is a common but meaningless organizing principle. Admittedly, Leeloo is 
using reference software built for a purpose other than learning about the 
human race in one fell swoop, but for it to be useful as a reference, multiple 
ways of getting at the information — including some sense of information 
hierarchy — would be appropriate. 


Information designer Richard Saul Wurman identified five 
primary ways to organize any set of information: by category, 
time, location, continuum (along a variable), and alphabet. 

Learning 239 

The first four of these add a level of meaning that helps a learn- 
er compare and make sense of the information, while the last 
is arbitrary. Alphabetic organization was popularized because 
most English speakers have the order memorized, but mod- 
ern search capabilities obviate such manual search methods. 
So, while it's best to give learners ways to reorganize as part 
of making sense of the information, the smart default is most 
often anything but alphabetical organization. 

Another (and much beloved) example is The Hitchhiker's Guide to the Galaxy, 
a small device with a horizontal screen and a metal frame that folds in half 
like a book. To use it, the reader opens it and speaks a word or phrase. In 
response, an animated and narrated clip describing the topic plays on the 
screen. The characters use the Guide to look up a variety of things during 
the course of the film, including information on the Vogon alien race, the 
language-translating Babel fish, and even practical matters such as how to 
fill out the notoriously bureaucratic Vogon forms (Figure 11.21). 

A device seen in Apple's industry video, Project 2000, gives us some insight 
into how speculative interfaces can act as a real-time reference. In one 
vignette, a man is learning to read with a device. It listens to his words, 
highlights the words as he's speaking them, and even calls up the proper 
pronunciation when he's unable to read the word correctly (Figure 11.22). 

FIGURE 11.21a-c 

The Hitchhiker's Guide to the Galaxy (2005). 


Chapter 11 

AAa . i.i yst npi *' * 

FIGURE 11.22a-c 

Apple's Project 2000 (1988). 

There is one other aspect of this inspired prototype that bears mentioning. 
During the course of the lesson, the man tires of the instructional book he's 
assigned. Instead, he circles an article in the newspaper's sports section, 
tells the device that he wants to read "this," and places it face down on the 
screen. The device scans the article and automatically converts it to text, 
allowing him to read in the same way as before, but with his own material. 


When a learner is focused on acquiring a skill, he or she will 
be simultaneously engaged in the content of the task. For 
example, learning to sing is partly motivated by a love of the 
song being sung, not just getting arbitrary notes correct. Give 
learners options of many types, or if feasible, the ability to 
supply their own content to help keep them interested and 
engaged (see Figure 11.22). 

Machines to Think With 2 

Information must be internalized in the mind of the learner before it can 
become knowledge. Internalization isn't a simple process. Learners need 
tools to consider new information in different ways, compare it to what they 
already know, build new representations, and consider new hypotheses. 
For skills learning, the practice interfaces seen above are sufficient to this 
task, but for more abstract and symbolic information, learners need a place 
to ruminate on symbols and concepts, as well as systems to make sense of 
what they study. 

The tools and processes for abstract learning must be, necessarily, open- 
ended, and adaptable. Pencil and paper, clay, drawing boards, and Legos are 
all good examples. In the survey we see such tools being underused, adapted 
for unintended purposes, or in use only in the background. 

2 The title of this section is taken from a chapter in Howard Rheingold's Tools for Thought, 2000 
(Cambridge, MA: MIT Press). 



In the "Mirror, Mirror" episode of the original Star Trek TV series, Captain 
Kirk consults the computer to learn whether a poorly understood accident 
could have been produced deliberately (Figure 11.23): 

KIRK: Computer. 

computer: Ready. 

kirk: This is the Captain. Record: Security research, classified 
under my voiceprint or Mr. Scott's. 

computer: Recording. 

kirk: Produce all data relevant to the recent ion storm, correlate 
following hypothesis. Could a storm of such magnitude 
cause a power surge in the transporter circuits, creating 
a momentary inter-dimensional contact to a parallel 

computer: Affirmative. 

KIRK: At such a moment, could persons in each universe in the act 
of beaming transpose with their counterparts in the other 

computer: Affirmative. 

kirk: Could conditions necessary to such an event be created 
artificially, using the ship's power? 

computer: Affirmative. 

FIGURE 11.23 
Star Trek: The Original 
Series, "Mirror, Mirror" 
(Season 2, Episode 9, 


Chapter 11 

FIGURE 11.24a-c 
Starship Troopers (1997). 

What's notable is that during the interchange, Kirk isn't playing with 
ideas, he's just asking yes or no questions. Even though he's asking about 
something fairly mind-blowing (which could have changed the nature of 
the Star Trek franchise into something much more like Quantum Leap), he 
is essentially using the device as a reference. Still, the scene gives the sense 
that this is something that has never been done before, with the computer 
instantly modeling options and testing variables to produce its answers. 
This makes it a good tool for thinking. 

In one of the schoolroom scenes from the film Starship Troopers, Rico makes 
an animated drawing of him and Carmen about to kiss, using his pen and 
tablet. He starts by drawing their profiles in white lines. He then adds some 
flat color and animates them such that the faces get closer, their eyes close, 
and their mouths open in readiness of a kiss. He then sends it to her through 
an in-class message system. She sends it back after adding a funny bubble 
gum bubble that ruins the anticipated kiss (Figure 11.24). 

Though this is used in a social way, the tool, called FedPaint, is provided 
by the school and seems to be designed for use in class to draw and model 
events across time rather than flirting. 

Why do we see so little of these most important tools? Most likely, the needs 
of the story outweigh any need to generate novel ideas. Because of the time 
constraints in cinema and television, writers prefer to show the sudden 
insight and eureka moment to the slow and sometimes disorderly process 
of real learning. Additionally, dialogue is a natural relationship between a 
teacher and a student, creating a learning-moment bias toward characters 
interacting with each other through technology. 



FIGURE 11.25a-c 

Star Trek IV: The Voyage Home (1986). 

Testing Interfaces 

Testing is seen in the survey a few times, all within the Star Trek films. 
In Star Trek IV: The Voyage Home, we see such a system when the recently 
reanimated Spock is rebuilding his sense of self and knowledge of the world. 

In the testing cell, he approaches a bank of three transparent screens. When 
he says, "Computer, resume testing," a metallic voice begins to ask him 
questions on a wide variety of esoteric topics, such as, "Who said 'Logic 
is the cement of our civilization with which we descend from chaos using 
reason as our guide?'" 3 and issuing challenges such as "Adjust the sine wave 
of this magnetic envelope so that anti-neutrons can pass through it but anti- 
gravitons cannot." When Spock correctly answers a question, the computer 
responds with "Correct!" and moves on (Figure 11.25). 

For each question, either the text or an illustration of the posed problem is 
presented on the screen and remains there until Spock answers it. He answers 
some of the questions vocally. For others, he places his hands on a set of touch 
pads. Only his gaze identifies which of the three simultaneous questions 
he is answering. He answers with increasing rapidity until he comes to a 
particularly difficult question he does not understand (Figure 11.26). 

3 T'Plana Hath, matron of Vulcan philosophy, in case you were wondering. 


Chapter 11 

FIGURE 11.26a,b 

Star Trek IV: The Voyage Home (1986). 

Seriously, he's stumped. 

One of the most memorable examples of testing interfaces is in the Star 
Trek reboot film when a young Spock is attending school on Vulcan. He 
is standing at the base of a concave hemisphere that surrounds him with 
a projected expanse of overlapping and moving images, formulae, and 
illustrations (Figure 11.27). He responds to a voice asking him factual 
questions, such as "What is the formula for the volume of a sphere?" As he 
answers questions correctly, the related figure fades from view and another 
is asked. We do not see Spock make an error, so we don't know what would 
happen in that case. 

When the camera pulls back to reveal many similar learning pods with one 
student at the center of each, we understand that testing on Vulcan is done 
alone, and we assume that each student progresses through this gauntlet at 
his or her own pace. 

In both cases, the interface fires a barrage of questions at the student, 
testing recall of facts in rapid succession. 


1 *?i A 




FIGURE 11.27a-c 
Star Trek (2009). 



Both of these systems equate intelligence with the simple recall of facts. 
Memorization is a core skill, but data in the age of the Internet is cheap. 
Just as or more useful is the ability to apply that knowledge— to take a 
complicated problem in the real world, identify what information is and isn't 
pertinent, and form and execute a plan for solving it. Granting the benefit 
of the doubt, perhaps these testing interfaces we see are just one part of 
Spock's education and there are other systems for learning other skills. It 
makes sense that a filmmaker would want to pick the most cinemagenic of 
possible learning components. This testing pod qualifies for its pace, exciting 
visuals, and an emotional callback to what it felt like for the audience to take 
difficult tests in their youth. 

Another testing interface seen in the Star Trek franchise is the Kobayashi- 
Maru test. To participate, a group of Starfleet cadets gather on a simulated 
starship bridge in assigned roles. The test taker plays the role of captain. 
Through video screens and information interfaces, the crew encounters a 
situation in which they must face hostile and overpowering enemies while 
trying to rescue the crew of a stranded vessel. Though the cadets try to solve 
the problem, it is designed to be unbeatable, and is rather a test of character, 
creativity, and the ability to handle stress. 

We first see the test in Star Trek II: The Wrath of Khan, with Cadet Saavik in the 
role of captain (Figure 11.28). It next appears in the 2009 Star Trek reboot film, 
in which James T. Kirk reprograms the simulation so that he can easily defeat 
it— an event referred to in Star Trek several times but never before shown 
(Figure 11.29). 

FIGURE 11.28a-d 

Star Trek II: The Wrath of Khan (1982). 


Chapter 11 

FIGURE 11.29a-d 
Star Trek (2009). 

Both versions of this test illustrate an interface that simulates a complex 
problem in a very realistic way, requiring the cadet to analyze the situation, 
form a plan of action, and execute it to see the results. The only thing to 
break the highly realistic illusion of the simulation is the observation gallery 
out of view of the cadet in the captain's chair. We know from dialogue in the 
films that, though it's rare, cadets may take the test multiple times, enabling 
them to learn from the experience, in order to reflect on their actions or 
reconsider how to handle the situation, particularly their emotions. 


The fear of failure and the consequences of failure can paralyze 
a learner. Grant them the confidence to try new skills and new 
approaches by providing a safe space to learn, where the con- 
sequences aren't permanent, there is an opportunity to reflect 
on performance, and it is easy to return to face the challenges 
again and improve. 

Case Study: The Holodeck 

With the exception of direct brain downloads, every type of learning 
interface in sci-fi is found in the holodeck from the Star Trek franchise. 
Though references cite its original appearance as "the rec room" in Star Trek: 
The Animated Series, the much more popular Star Trek: The Next Generation 
brought it to a wide audience. 



The holodeck is an amazingly ambitious piece of speculative technology. It is 
a chamber that can project perfect volumetric projections of any conceivable 
scenario, including the environment, objects, and lifelike characters 
within it. The environments and objects are perfectly visually detailed. The 
characters within it behave with full agency and even occasionally display 
a disturbing degree of self-awareness. It also creates finely controlled force 
fields to provide users with real-life tactile feedback and various degrees of 
real-world physics. 

Outside the chamber is a wall-mounted touch interface that lets users see if 
the holodeck is occupied, schedule their own session, and select a holodeck 
program. While in the room itself, the holodeck is controlled primarily by 
voice. Users speak aloud, addressing the computer with such commands as 
"Computer, halt program" or "Computer, freeze." The simple voice control 
means the barrier to participation is minimal. Users can also summon 
command panels in the doorframe for additional control and instruction. 

In the holodeck, users can interact with notable fictional or historical figures 
and places, play virtual sports, and ride virtual horses across limitless 
landscapes. The user's appearance can be altered, though only in subtle ways, 
such as color saturation or simple changes in clothing. The crew can reserve 
the room and use it for a huge array of purposes: entertainment, games, 
training, problem solving, exercise, sex, and even authoring holodeck novels. 

For learning, in particular, the holodeck is used in a number of ways. 

Psychomotor Training 

One of the first examples of the holodeck being used for psychomotor 
training is when Lieutenant Yar demonstrates an Aikido program for 
visiting dignitaries (Figure 11.30). She summons the Aikido master and 
performs a few moves, explaining that the master evaluates your progress 
and adjusts accordingly, challenging the user to improve. When the 
demonstration is over, she dismisses it with a verbal command. 

FIGURE 11.30 
Star Trek: The Next 
Generation, "Code 
of Honor" (Season 1, 
Episode 4, 1987). 


Chapter 11 


Learners can only get so far studying on their own, and they 
won't add significantly to a culture's body of knowledge if they 
are rediscovering lessons that are already well documented. 
To help learners move to the forefront of knowledge, systems 
should be able to provide some expert guidance when asked 
(see the section on "Degrees of Agency: Autonomy and As- 
sistance," page 190, in Chapter 9). This can be as simple as 
reasonable goal formation. For example, if a student tells the 
computer "I want to be the best pilot in the Alpha quadrant," 
then the computer should be able to suggest a course of study 
and practice to achieve that goal. 

System guidance and assistance can also challenge assump- 
tions. If in the "Once Upon a Time" episode that opened the 
chapter, Naomi had shouted, "Oh no! Flotter's been killed!" a 
guide could have asked, "Are you sure? Let's watch it again. 
What do you see?" and by pointing out the evaporation, set 
her down the right path. (Read more about this particular 
example below.) Often, this type of guide should be outside of 
the main narrative to be able to discuss things dispassionately 
and without confusing the learner. 

Other skill training includes very specific field rehearsals. In the episode 
"Chain of Command (Part 1)", an awayteam rehearses their mission in 
a perfect replica of Kardassian tunnels for a difficult covert operation 
(Figure 11.31). By rehearsing inside a replica, they can commit the geography 
and tactics to memory, pause and repeat the more difficult parts, and get 
their timing down to the second. 

FIGURE 11.31 
Star Trek: The Next 
Generation, "Chain of 
Command (Part 1)" 
(Season 6, Episode 10, 



FIGURE 11.32a-c 

Star Trek: Voyager, "Once Upon a Time" (Season 5, Episode 5, 1999). 


The infinite display capability of the holodeck means that the presentation 
of material can be tailored to the learner— in whatever genre or presentation 
medium suits the scenario best— aural, visual, or tactile. 

The Star Trek: Voyager episode mentioned at the beginning of this chapter 
("Once Upon a Time") involves a young girl's use of a holodeck novel that 
feels much more like a storybook for learning. Naomi enters a colorful 
world where charming characters embody materials from nature: Flotter 
represents water, and Trevis represents trees. Solving problems for these 
characters teaches her about their properties, and she is deeply engaged 
through the narrative presentation. 

A major problem occurs when she and her friends, Flotter and Trevis, encounter 
the Ogre of Fire, who in his fiery rage evaporates Flotter (Figure 11.32). The 
narrative creates a problem that Naomi wants very much to solve— how to get 
Flotter back. To do so, she has to learn how a liquid turned into a gas through 
evaporation can be condensed back into water again. 


There is no reason why the holodeck could not be used to present reference 
material in a highly engaging way, but this is not seen. Occasionally, 
characters may ask the computer factual questions, but there is no 
Hitchhiker's Guide to the Galaxy-type holodeck programs that appear so 
the learner can conduct some research while the main holodeck program 
is on hold. Instead, the holodeck facilitates simulation and learning 

Still, Naomi has access to reference materials through a computer terminal in 
her cabin, which she uses to solve the problem of her evaporated friend Flotter. 

Though the details of this reference interface are not shown, it is vital that she 
have access to the materials so that, in solving the problem, she can learn. 


Chapter 11 

Machines to Think With 

The storybook presentation of "Once Upon a Time" is itself a way of thinking. 
Characters present problems, such as "Where do you think the fire came 
from?" and Naomi offers hypotheses after thinking about it for a bit. 

There are other ways to use the holodeck to represent current thinking 
and evaluate it, of course. In the Star Trek: The Next Generation episode 
"Nth Degree," Lieutenant Barclay has his brainpower greatly amplified by 
an unknown force. Unable to sleep, he spends the night in the holodeck, 
discussing ideas with a holodeck display of Einstein. They write equations on 
a chalkboard, then discuss and revise them (Figure 11.33). 

Evaluating hypotheses means considering multiple options. In the Star 
Trek: The Next Generation episode "Booby Trap," Geordi La Forge uses the 
holodeck for just this purpose. Faced with a life-threatening problem, he 
calls up a virtual recreation of the original designer of the Enterprises 
engines to work through the possibilities. Together, they create simulations 
to vet their ideas and find the best solution (Figure 11.34). 

FIGURE 11. 33a, b 

Star Trek: The Next Generation, "Nth Degree" (Season 4, Episode 19, 1990). 

FIGURE 11.34a-c 

Star Trek: The Next Generation, "Booby Trap" (Season 3, Episode 6, 1989). 



La Forge employs an educated hit-or-miss method with the simulation. For 
example, he asks, "What is the effect of reducing thrust levels another four 
percent and adjusting trajectory to compensate when in an energy-draining 
environment?" Though this just maybe his personal style of exploration, the 
interaction would be more efficient if it simply tested a range of options and 
shared those that best fit his criteria. 


Hit-or-miss methods of testing hypotheses are a waste of a us- 
er's time. When computing power is abundant, let the comput- 
er do as much look-ahead computation as it can, present the 
results, and guide users proactively toward the best options. 

This example illustrates another lesson: recognition is easier than recall. 
La Forge is having to watch variables, represented as horizontal bars, 
change over the course of the simulation. But when is the variable at its 
highest? When is it at its lowest? The display forces him to remember. Had it 
presented a line chart of the variable over time with a highlight showing the 
current state of the variable, he would not have to recall how it has changed 
and additionally understand it in context. 

A second way this principle could have improved this interface is to help 
La Forge compare one model against another visually. What was this 
same variable doing in the other simulation at this point? Was it better or 
worse? What about the end results of the multiple simulations? How did 
this simulation compare against the one four variations back? Presenting 
real-time comparisons between simulations and keeping results persistently 
visible would help lighten his memory load and help him recognize the best 
solution when he came across it. 

Though the holodeck could conceivably handle much more complicated 
ways of representing an abstract hypothesis, such as multidimensional 
graphs of equations, nothing like this is shown in the holodeck. 


Recall places a burden on a user's short-term memory, which is 
more fallible than the ability to select from a set of visible op- 
tions. Whenever possible, display options or data across space 
to let users review, feel confident that they have identified the 
salient option, and act on it. 

252 Chapter 11 

Lessons Unique to the Holodeck 

The holodeck's unique capabilities provide some lessons for learning systems 
not found elsewhere in the survey. 

At the end of "Once Upon a Time," Flotter even expresses admiration over 
the way that Samantha, Naomi's mother, has grown since he saw her last. 
This social interaction indicates that the holodeck characters remember 
individual learners and can respond to how they change over time. 


Learning is not simply a matter of stringing together a series 
of small lessons. At times, it is helpful to step back, create 
context, and put the pieces together to understand something 
of the whole. Additionally, recognizing the progress that they 
have made gives learners a sense of accomplishment. For this 
to happen, the system needs to remember and recognize an in- 
dividual learner. Though it is handled conversationally, it is clear 
that individual programs recall individual users and can return 
them to the point where they last left off. 

In the "Author, Author" episode of Star Trek: Voyager, crew members use the 
holodeck to read the Doctor's holographically recorded novel, in which the 
reader acts as the lead character in a number of scenarios. Each scenario 
is meant to elicit empathy for that character's oppressive circumstances. 
Because the lead character is based on the Doctor's own experiences, the 
stories become an interface to understand his perspective. 


If the system recognizes individual users, then they should be 
able to learn from each other's experiences, either in aggregate 
(e.g., "Most of your friends took the left path, Naomi. What do 
you think?"), or from direct observation of familiar situations 
from others' viewpoints. 

Another application of the holodeck is as therapy and social practice, 
creating a "trial version" of the real world. In the The Next Generation episode 
"Hollow Pursuits," Lieutenant Barclay uses the holodeck to work through a 
number of personal issues, modeling exaggerated versions of crewmembers 
with which to enact fantasies of romance or even domination. When the 
crew discovers these simulations, they become upset. Counselor Troi 
defends the program against deletion and encourages Barclay to use it as 
part of an ultimately successful therapy. 

Learning 253 


People are emotional creatures, and many of our lifelong les- 
sons are about understanding and dealing with our emotions. 
While many skills and knowledge are impersonal and "extro- 
spective," future learning technologies must account for these 
more introspective and interpersonal topics as well, through 
instruction, role-playing, and the ability to retry situations that 
don't play out as well as expected. 

What We Don't See 

Even with the holodeck representing the best of the learning interfaces, 
some tools and features we don't see in it would help a learner even more. 



When users practice difficult physical skills, the holodeck could 
provide gentle force field nudges to act as a scaffold. Although 
it could do this with character embodiments, such as a trainer 
to help with acrobatic maneuvers, it does not need to be con- 
strained to the physical limitations of a humanoid to provide 
this service. A diver rehearsing a back one-and-a-half som- 
ersault with four-and-a-half twists could never expect a real 
trainer to help position her correctly throughout the dive, but 
the holodeck could, while providing feedback on the tactics 
and principles behind the assistance. It could even simulate the 
slowing of time to allow the diver to concentrate on form. 



Instructors in the real world often point out details of a thing 
while reviewing it. The holodeck could easily do this by aug- 
menting its real-world presentation with a 3D augmented real- 
ity overlay of relevant metrics and useful information similar to 
a heads-up display. For example, while Naomi is reviewing the 
Flotter incident (see Figure 11.1), she might want to reference a 
diagram of the water cycle, with its components labeled in the 
scene, such as "heat source," "water," and "water vapor." (See 
Chapter 8 for more on augmented reality.) 

254 Chapter 11 



Though the holodeck does have a memory of its past users, 
where are the tools to encourage social learning? We don't see 
collaboration by multiple users in separate holodecks or on dif- 
ferent ships, which would allow students to study together and 
help each other. We don't see friendly competition among us- 
ers of popular programs, such as a leaderboard for completing 
a mountain climb within time and adhering to safety protocols. 
And learners are surrounded by other individuals— such as 
family, friends, teachers, and counselors— who are not directly 
involved with the learning, but would like to know how things 
are progressing. Where does Samantha go to see Naomi's ac- 
complishments and challenges? 

Learning: Aiming for the Holodeck 

People in the real world are constantly learning, and technology has 
advanced how interfaces can help learners see, understand, model, and test 
their new skills, both physical and mental. 

But learning technology seems to be a mismatch for screen-based sci-fi. 
Certainly characters learn, but they do it either through events in the story 
or through student-teacher relationships. Using purely technological tools 
isn't as cinemagenic to watch. 

As a result, learning interfaces in sci-fi generally come up a bit short. They 
neglect cognitive modeling and testing (machines to think with), offer a 
narrative shortcut to the messy process of actual learning (direct downloads), 
showcase tools for developing physical skills (psychomotor practice), or play a 
strong role as a narrative tool even if their use in real-world learning would be 
small or dubious (testing and presentation and reference tools). Perhaps when 
learning interfaces in the real world become more cinemagenic, they'll make 
their way into sci-fi, and we can learn more from them. 

Learning 255 



Assistive Medical Interfaces 259 

Autonomous Medical Interfaces 280 

Life and Death 283 
Sci-Fi Medical Interfaces Are Focused Mainly 

on the Critical Situation 290 

FIGURE 12.1 
Star Trek: The Original 
Series, "The Naked 
Time" (Season 1, 
Episode 4, 1966). 

In sickbay, Dr. McCoy gestures with an open palm, inviting Mr. Spock to 
rest against the nearly vertical biobed. Nurse Chapel presses down on 
the head of the bed, swiveling it horizontally. At once a panel above the 
bed illuminates, showing how the patient's vitals are doing against normal 
parameters (Figure 12.1). McCoy glances at the panel and shakes his head, 
saying to Spock, "Your pulse is two hundred and forty two, your blood 
pressure is practically nonexistent . . . assuming you call that green stuff in 
your veins blood." 

Unperturbed, Spock sits up and replies, "The readings are perfectly normal 
for me, Doctor, thank you. And as for my anatomy being different from yours, 
I am delighted." Having completed the examination, the nurse swivels the 
biobed back to its near-vertical position, allowing Spock to easily step off. 

Medicine is a complex domain that deals with complicated, connected, 
and overlapping biological systems: skeletal, digestive, muscular, 
lymphatic, endocrine, nervous, cardiovascular, reproductive, urinary, and 
psychological. The number of things that can go wrong with any of these 
is huge. Medical practitioners must attend medical school for years to 
fully grasp the knowledge needed to master their chosen specialty. Neither 
Hollywood nor audiences have that kind of time, which puts a major 
constraint on medical interfaces in sci-fi. 

What kind of time is available to tell satisfying stories involving medicine in 
sci-fi? It depends on the medical literacy of the audience. The goal is to strike 
the right narrative balance between boredom and unintelligibility. At the 
most remedial level, according to Harvard psychologist Steven Pinker, all 
people have a basic biological understanding of life: "All living things possess 
an invisible essence that gives them their power, drives their growth, and is 
inherited by their progeny. A dead thing no longer possesses this invisible 


Chapter 12 

essence." 1 Modern sci-fi audiences probably come to the cinema or turn on 
the television with a little more medical expertise than this, but in general, 
Hollywood tends to err on the side of caution and present medical problems 
simply— that is, as "dire situations," with any interfaces in the scene helping 
to build the tension and explicate final outcomes. 

This chapter is primarily organized around the two types of medical 
interfaces: those that help people perform medicine, and systems that 
perform medicine autonomously. These sections are followed by a discussion 
about technology assisting birth and signaling death. 

Assistive Medical Interfaces 

Most of the medical technologies we see in the survey help a human perform, 
or as in the next section avoid, medicine. 

An Ounce of Prevention 

Western medicine focuses much more on treatment than prevention, but 
prevention helps people avoid the pain and stress of problems in the first 
place. One such example appears in the Battlestar Galactica TV series 
reboot. In season 3, pilots must steer their ships through areas with high 
levels of radiation while searching for food supplies. To help them gauge 
the amount of exposure they've received, they wear a badge on their wrists 
that slowly turns black as it is exposed (Figure 12.2). When the badge is 
completely black, the pilot is at the maximum safe radiation dosage. 

There are few examples of preventative interfaces in the survey. There are 
some plots involving vaccines and antidotes, but they are usually given as a 
treatment rather than as prevention (see the section on injections, below). 

FIGURE 12.2a,b 

Battlestar Galactica, "The Passage" (Season 3, Episode 10, 2007). 

1 Pinker, S. (2002). The blank slate: The modern denial of human nature (pp. 220-21). 
New York: Viking. 





Preventative interfaces could help warn users away from be- 
havior or circumstances that would result in medical problems, 
allowing for just-in-time notifications and an easy-to-judge 
comparison of outcomes. There are many nascent technology 
projects like this happening in the world of health care now, but 
sci-fi seems disinterested in them. 


The first task for medical professionals is to understand the problem that 
they're up against. Interfaces for evaluating patients break down into 
technology for monitoring, scanning, and testing. What distinguishes 
each? For the purposes of this chapter, monitoring is the measurement and 
display of real-time, specific physiological data, such as heart rate. Scans 
are the visualization of large areas of general physiological data, such as 
an MRI (magnetic resonance image) or X-ray. Tests are the measurement of 
specific, discrete physiological data, such as checking for the presence of a 
certain antibody. These tasks may be done repeatedly throughout a patient's 
diagnosis and treatment. 


Most medical interfaces seen in the survey belong to the first category: 
monitoring the health of a patient. These interfaces help visualize the story 
of a character's internal health status — which is especially useful when the 
producer doesn't want to bloody or disfigure a protagonist to show that 
there's a problem. 

What gets monitored? In modern medicine, there are seven biometric 
indicators commonly used to monitor patients. These are heart rate, arterial 
blood pressure, central venous pressure, pulmonary artery pressure, 
respiratory rate, and less commonly, blood oxygen and body temperature. 
These seven indicators tell physicians and nurses a lot about a patient's 
current health status within the context of their illness. 

In the survey, monitoring interfaces almost always include some variant 
of these basic vital signs shown as waveforms scrolling from right to left. 
Two aspects of monitoring interfaces are nearly universal: First, when only 
a single waveform is shown, it is almost always heart rate. Second, when 
things become dire, elements of the interface turn red and alarms sound to 
draw the physician's (and the audience's) attention to the problem. Beyond 
these two similarities, interfaces vary a good deal. 

In Star Trek: The Original Series, patients' vital signs are monitored on a screen 
built into the wall above each biobed in sickbay. This screen features an array 
of six graduated registers with white triangles indicating whether a reading 

260 Chapter 12 

FIGURE 12.3 

Star Trek: The Original 

Series, "Space Seed" 

(Season 1, Episode 22, 


is in a stable, concerning, or dangerous range. Readings are labeled with 
things like temperature, blood Q3 levels, and "cell rate" (whatever that means). 
"Normal" health is indicated in green in the small range around the midpoint, 
transitioning above and below to yellow and then red (Figure 12.3). 

This monitor is useful for showing the current state of the patient, but it 
doesn't indicate trends. Its design poses a number of other challenges as 
well: the label species and the data field humanoid look too similar, pulse 
and respiration are shown without metrics at all, and numbers are too small 
to be read at a distance, to name just a few. 

One of the best features of the interface is that when there is only one patient 
on a biobed in sickbay, his or her heart rate is represented as audio as well. 
This benefits the audience, of course, but is also helpful ambient information 
for doctors and nurses, especially in critical situations like surgery. Having 
an auditory indicator allows them to focus their eyes and hands on other 
tasks while being aware of the status of this fundamental sign. This auditory 
signal disappears when there is more than one patient in sickbay, providing 
an excellent lesson in system design. 


Many systems are built to handle any number of items simul- 
taneously: zero, one, or more than one. When the system has 
zero items, its interface can shift to tools that allow for moni- 
toring or selection. When the system has multiple items, its 
interface must allow the user to indicate which of the items is 
currently selected and should be acted on. Ambient signals, 
such as background color and audio, would be difficult to 
associate with one of the particular items. 



When there is only one item, however, the system should adjust 
accordingly. No one should have to select when there's only 
one option. Even a simple shift of language to confirm the 
selection accomplishes this. Additionally, selection tools for the 
primary object are no longer necessary, and ambient signals 
can be used since their connection will not be ambiguous. De- 
signers should adjust the designs of systems to accommodate 
the special case of having only one item. 

Having noted how the audible heartbeat assists the doctors and nurses, 
we should also note that there is no indication about how this same signal 
affects patients in the sickbay. Does it cause them stress, or fill them with a 
morbid self-awareness? Unless the ambient signal was delivered only to the 
medical professional by augmented reality systems, care would need to be 
taken to consider all stakeholders of a design. 

Shifting to Waveforms: Data over Time 

The now-familiar waveforms of the electrocardiograph, or EKG, have been 
known to medicine since Willem Einthoven invented the device around 
1903. But it was not until 1932 and the development of direct pen on paper- 
rather than photographic— instruments that the technologies began to be 
used widely. Starting in the 1940s, people would have been exposed to these 
displays, but they aren't depicted in sci-fi until 1968, in 2001: A Space Odyssey. 
This film showed vital signs on computer screens as the HAL-9000 computer 
monitored the hibernating crew members (Figure 12.4). Waveforms were 
commonplace in the world at the time, but the fact that they were on a 
computer screen with dynamic displays and smart alerts brought the notion 
of computerized patient monitoring into the future. 

2001 and most of the sci-fi that came after it augmented the display of vital 
signs with big labels to tell the audience in words what might be missed 
from the graphics, and what a doctor who is knowledgeable about the data 
might interpret from a glance at the same interface. 

FIGURE 12.4 

2001: A Space Odyssey 


262 Chapter 12 


People instantly identify waveforms as vital signs. In addition to 
their instant recognition, waveforms are well suited to the task 
of reading a simple variable over time within a range of values, 
allowing physicians to quickly understand trends and identify 
problems. For these two reasons, designers should stick with 
displaying vital signs as waveforms unless there is a very good 
reason to depart from this standard. 

After 2001, we see many similar medical monitors in sci-fi that differ mostly 
in their visual style (Figure 12.5). 

Some sci-fi medical monitoring interfaces are very complex. As an audience, 
we maybe impressed with a character's mastery of such complexity, but it's 
likely that these systems wouldn't be useful in the real world. The multiple, 
overlapping layers seen in Star Trek seem to obscure important data more 
than highlight it (Figure 12.6). In these cases, films must show bright 
overlays or use character dialogue to compete with the background noise 
of their own designs and let audiences know what's going on. 

■tJ*" lull—- 

' zJm 

1 ^^^W 

FIGURE 12.5a-d 

The Island (2005); Aliens (1986); Space: 1999 (1975); Defying Gravity (2009). 



FIGURE 12.6 
Star Trek (2009). 


Many of the medical monitoring interfaces that we see in sci-fi 
are meant to impress us quickly with motion and complexity, 
but they defy usability and readability principles of clarity. For 
such systems in the real world, this kind of showiness would 
become noise, diluting the user's attention. Designers should 
show just enough information to provide the context, and 
should control the visual design so that directing the physi- 
cian's attention to problems is easy. 

The TV series Space: 1999 introduced medical monitoring concepts that 
were novel for its time. Each resident in the space station was continuously 
monitored by an "X5 computer unit." When some medical alert occurred, the 
system would automatically contact Dr. Russell and her medical staff in the 
sickbay with both on-screen text and an audible alert. The system had voice 
recognition input and provided text output that it would simultaneously 
speak, two concepts that were very new at the time. 

In the episode that shows this ubiquitous monitoring and alert system, 
technician Dominix's death was nearly instantaneous, so the X5 didn't have 
time to issue any critical warnings to summon an emergency response team, 
but it seems safe to imagine that such a system would be capable of this. 

Additionally, the X5 could make medical "conclusions" based on data it had, 
though this fell short of actual diagnosis. In the pilot episode, Dr. Russell 
consults the X5 by voice about a particular patient she has been monitoring. 
She points her comlock (a portable communications device) at it, presses a 
button, and says, "Computer, please verify that last report." The X5 replies, 
"Stage 5 mutation complete. All brain activity stopped. Cell life sustained 
by artificial life support systems only. Conclusion: Astronaut Eric Sparkman 
deceased" (Figure 12.7). 

264 Chapter 12 

FIGURE 12.7a,b 

Space: 1999, "Breakaway" (pilot episode, 1975). 

Suspended Animation 

Suspended animation is quite common in sci-fi, and often includes a special 
interface for onlookers to monitor the health of the suspended. In Star Wars 
Episode V: The Empire Strikes Back, Han Solo is encased in "carbonite" for 
transport by the bounty hunter Boba Fett. For the procedure, Han is lowered 
into a chamber and the process encases him in a slab of solid material that 
includes a small control panel on the side to monitor his health and, in the 
next film, revive him with the press of a few buttons (Figure 12.8). 

More recent suspended-animation monitoring interfaces adopt more 
modern styles, such as the real-time 3D organ rendering of Minority Report 
and the cool monochrome otPandorum (Figure 12.9). 

FIGURE 12.8a,b 

Star Wars Episode V: The Empire Strikes Back (1980). 

FIGURE 12.9a,b 

Minority Report (2002); Pandorum (2009). 




Sometimes a physician needs to scan a patient to create a picture of the 
patient's insides. Many of these scans are visually similar to an X-ray. Most 
of these are static, fixed pictures, but some show "live" data changing in real 
time. In almost all cases, patients are lying down and covered by the reading 
technology (Figure 12.10). 

The scanning interface from Dune is notable because it displays a life-size 
image in real time, but the screen is significantly smaller than the patient's 
body. To examine different parts of the body, the doctor slides the screen 
left and right, and the screen adjusts to show the part of the body beneath it 
(Figure 12.11). 

FIGURE 12.10a-c 

Space: 1999 (1975); Star Trek: Voyager 

(1995); Alien (1979). 

FIGURE 12.11 
Dune (1984). 


Chapter 12 


Users have years of direct experience with manipulating 
objects in the physical world. Designing physical controls for 
manipulating systems reduces the amount of training needed 
and the amount of cognitive weight during use, where the 
physical movement is natural to the intended result. This lets 
the user focus attention on the more difficult problems at hand. 
This effect is increased when there is a direct mapping of the 
controls— for example, moving the display screen left and right 
to move the X-ray camera similarly. (See Chapter 2 for more on 
physical controls.) 

3D Visualizations 

The biggest leaps forward in the types of visualizations for monitoring 
coincided with advances in computer-generated graphics in the 1990s, 
which enabled the display and animation of such internal systems as bones, 
nerves, and organs. These visualizations help show physical, immediately 
apparent problems such as fractures, tumors, or internal bleeding, and so 
are very useful for physicians. 

However, 3D visualizations still need to be augmented by 2D monitoring 
graphics and waveforms. This makes sense because the value of the EKG-like 
readouts is precisely that they show vital information invisible to the naked 
eye. For instance, even if you had lovely, full-color, selective X-ray vision into 
the human body, what organ would you look at to determine your patient's 
blood pressure, or how it has been trending over the last two minutes? 

In addition to being useful for the physician, these "transparent human" 
visualizations are very cinemagenic, which is one of the reasons they have 
appeared with more frequency in recent years. 

A good example comes from Star Trek: The Next Generation. In the episode 
"Ethics," Dr. Crusher consults with Dr. Russell about an experimental spinal 
surgery for Worf. They use a volumetric display of his spine to aid their 
discussion (Figure 12.12). 

In the movie Lost in Space, Judy's suspended animation tube fails to revive 
her. She's placed on a medical bed that projects a translucent, real-time 
volumetric projection of her internal organs above her and that slowly lowers 
to overlay her body. A pink, translucent cube rotates around her heart to tell 
onlookers where the problem is (Figure 12.13). A separate screen displays 
vital signs. Using the bed's interface, Dr. Smith attempts to resuscitate her 
with what amounts to a sci-fi crash cart. 

Medicine 267 

FIGURE 12.12 
Star Trek: The Next 
Generation, "Ethics" 
(Season 5, Episode 16, 

FIGURE 12.13 

Lost in Space (1998). 

As shown in the movie, Judy's volumetric projection works beautifully to let 
us see through her skin to her internal systems. This is good, because it's a 
noninvasive way to understand what might be wrong in a physical context. 
But it can't work the way it appears at first glance. Consider this: How 
does the display appear for Maureen, Penny, and Dr. Smith, who are all on 
different sides of the bed? Remember that the projection of her heart appears 
to be inside her chest. This could work only if it were an optical illusion 
giving the appearance of depth from one particular observer's point of view. 
If that person were to walk around the bed, the perspective would be all 
wrong, like seeing one of those sidewalk chalk illustrations from the wrong 
direction (Figure 12.14). 


Chapter 12 

FIGURE 12.14a,b 

Chalk art by Edgar Mueller. 

So is this idea a nonstarter? Not entirely, if we use it as an opportunity for 
apologetics. It could work if the system provided everyone who is viewing 
the patient with a custom rendering. So although it looks like a volumetric 
projection (and every clue in the scene indicates that that's what the 
filmmakers meant), it can only work if it is actually augmented reality— 
views that are particular for the observer. No one is wearing special viewing 
lenses in the scene to provide the overlay, so perhaps the augmentation is 
being broadcast from projection points on the operating table directly onto 
their retinas (and, we must suppose, the camera). In any case, examining what 
at first seems broken gives us a useful insight into how it might work in the 
real world. 

The TV series Firefly featured a scene with a display initially similar to the 
one from Lost in Space. In it, the ship's doctor, Simon Tarn, is able to do 
some noninvasive investigation into his sister's brain using a holoimager 
at a sophisticated medical facility. Using this tool, he is able to peer into a 
translucent volumetric projection of a real-time scan of her brain and activate 
gestural controls to change the orientation of the projection to get a thorough 
view of things. Animated graphics of brain activity and other vital signs are 
provided along the side of the display for quick reference (Figure 12.15). 

FIGURE 12.15a,b 

Firefly, "Ariel" (Episode 9, 2002). 



This is remarkable among the examples because, first, the physician 
interacts with the display to look for problems, and second, it is an 
interesting combination of gestural and volumetric projections. His direct 
gestures allow him to "grab" the image of her brain, strip away the rest of the 
display, and turn the projection to explore it. 

Although these sci-fi 3D visualizations are cinemagenic, they lack tools 
vital to the actual work of conducting a scan, which is to find the problem. 
The Lost in Space interface automatically finds the problem (her motionless 
heart) and highlights it for Dr. Smith. That's useful when the system's 
confidence is high, but not for the trickier task of finding an elusive problem. 
Firefly's interface comes closer. It allows Dr. Tarn to isolate the brain from 
the other body parts and turn it to look for surface problems. The display 
could have gone further, subtly directing him to prospective problem areas, 
such as scar tissue. It could also have had view controls— different spectra 
filters for comparing magnetic resonance and radiographic views, cross- 
section controls to let him observe arbitrary "slices," or scale controls to 
let him magnify the view to look at details very closely. (See the Chrysalis 
example on page 280 for a surgical example of magnification.) 

It might be worrisome if real-world designers were only referencing these 
sci-fi interfaces, but fortunately, they're ahead of Hollywood in this matter. 
For example, the Stanford Radiology 3D and Quantitative Imaging Lab at the 
Stanford University School of Medicine develops techniques for visualizing 
radiological studies as 3D-rendered images while hiding organs and tissue 
not related to the problem under consideration (Figure 12.16). Although they 
require an enormous amount of processing power, today's workstations 
and even laptops can generate interactive visualizations in real time, often 
within minutes of an exam's completion. 

FIGURE 12.16a-d 
Medical imagery 
from the Stanford 
Radiology 3D 
and Quantitative 
Imaging Lab 
surpasses the 
examples we see 
in sci-fi. 


Chapter 12 


We don't see a lot of medical testing interfaces in the survey. Two examples 
help explain why. The first is the medical tricorder from Star Trek, the 
portable box with the wireless, all-purpose sensor that McCoy uses on away 
missions (discussed below). The second is the ubiquitous DNA-testing device 
in the movie Gattaca, which takes tiny samples of blood, skin, or hair and 
provides the identity of its "owner" (Figure 12.17). 

In the real world, testing as part of medical procedures is uncertain ("Well, 
this could mean a polyp or it could mean gallstones"), iterative ("That 
test didn't reveal anything; let's try another"), and specific (measuring 
something with a device built for the job). 

Uncertainty and iteration aren't very cinemagenic or exciting and can 
digress distractingly from the plot. The exception is the medical mystery 
format, in which this uncertainty is the plot. Sci-fi dabbles in this, especially 
in TV shows. "Bones" McCoy from Star Trek: The Original Series, the Doctor 
from Star Trek: Voyager, and Dr. Tarn from Firefly all face novel challenges 
on their respective spaceships. Particular episodes can focus on these 
characters and the mysterious medical problems they face. But even these 
shows run into problems with specificity. 

Specificity is challenging because tests often require specific devices with 
specific sensors, all of which takes time to explain narratively and represents 
yet another prop to build. Star Trek gets around this with the medical 
tricorder because it's an all-purpose device, but this solution doesn't help us 
with the reality of testing. 

In addition, if the story is too specific about the thing being tested, you lose 
the audience, who don't have the necessary medical expertise to understand. 
So what's often more important to sci-fi than the test results is the diagnosis. 
Supporting this argument, most sci-fi medical systems that perform tests 
output a diagnosis rather than numerical data. When they do that, it's 
better to categorize them as diagnosis interfaces, as below. This is why 
Gattaca 's DNA testers are problematic: they provide unerring identities, not 
interpretable measurements. 

FIGURE 12.17a,b 
Gattaca (1997). 

Medicine 271 



Until we actually have exhaustive, perfect diagnosis machines, 
we'll want to improve the testing interfaces that clinicians use 
to do their work. Because sci-fi doesn't really dive into this 
regularly or with rigor, this leaves a number of tasks for design- 
ers to imagine and design: 

• Selecting which among the thousands of tests available 
are appropriate to the circumstances at hand 

• Performing the test accurately 

• Understanding the results 

• Communicating the results to patients 

• Determining next steps, whether initiating treatment or 
going back for the next test 


Evaluation supports diagnosis— that is, using measurements and observed 
symptoms to form a hypothesis about a patient's problem. But diagnostic 
tools are problematic both in the real world and in sci-fi, which may account 
for the very few examples we see in the survey. 

Diagnosis in the real world is difficult for a number of reasons. Knowing 
what signs to look for in the first place is constrained by the physician's 
knowledge and subject to his or her specialist biases. The set of possible 
problems is simply enormous. Tests to confirm a diagnosis maybe 
inconclusive or imprecise, and even if they are noninvasive, may cause 
additional complications or be expensive. Finally, even given the best set of 
reliable data, the givens may not easily resolve into a single root problem. 

Diagnostic technology proves problematic for sci-fi as well, and not just 
because the characters are dealing with alien species or environments. 
When the story is a sci-fi medical drama, the most common structure is 
that of a mystery puzzle to solve. The symptoms are shown in scenes that 
cleverly distract the audience from the true cause, which the protagonist 
puts together in a dramatic moment that makes the audience look back 
and think, "Well, of course. It was there right in front of our eyes the whole 
time." In these narrative structures, if the computer simply spat out the 
correct answer in scene 3, there would be no reason to go on. Diagnostic 
technologies are counter to the purpose of such stories. 

If the story is not a medical drama but has need for diagnosis, it still proves 
problematic, because an accurate depiction of the complexities may not add 
to the story. Fortunately, sci-fi has all the high-tech, noninvasive sensors 
and computing power of its authors' imaginations. Authors can streamline 
a diagnosis so that the medical MacGuffin quickly and confidently keeps 
the plot moving. 

272 Chapter 12 

The original Star Trek TV series introduced portable and fast diagnosis 
with its medical tricorder. The ship's doctor, Leonard "Bones" McCoy, would 
wave the handheld scanner over a patient and simultaneously look at the 
handheld screen (Figure 12.18). Though the series never showed exactly 
what he was looking at, after a glance at the screen he would describe 
exactly what was wrong with the patient, even if the problem was obscure 
or the patient was an alien species. Though McCoy might have been viewing 
a screen very similar to the sickbay monitoring screens discussed above 
(see Figure 12.3), it is more likely that the device was also able to suggest a 
diagnosis based on its readings. 

Other series in the Star Trek franchise evolved the appearance and shrunk 
the size of the medical tricorder. (Figure 12.19) 

FIGURE 12. 18a, b 

Star Trek: The Original Series (1966); studio prop photo. 

FIGURE 12.19 
Star Trek: Voyager 



Few other sci-fi properties have presumed such portability and speed of 
medical diagnostic tools. Although the medical tricorder was largely a 
narrative tool rather than a serious speculative technology for assistive 
diagnosis, the notion of a portable and near-instantaneous diagnostic tool 
inspires medical technology inventors to this day. And that's not by accident. 
In one deliberate attempt to have sci-fi influence the real world, Star Trek 
creator Gene Roddenberry reportedly signed a contract with Desilu/ 
Paramount stipulating that anyone who can create a device that operates 
like a medical tricorder can use the already-popularized name for their 
device. In essence, he is offering nearly 50 years of in-show marketing to 
anyone who can live up to the vision. 



The survey shows a number of different hardware designs for 
diagnostic tools, but it is pretty much devoid of software inter- 
faces for aiding diagnosis. There are some diagnostic tools in 
the real world, such as MEDgle and WebMD, that are targeted 
at both patients and practitioners, so it's just an instance of sci- 
fi not yet finding an interest in it. 

One important tool used to assist diagnosis is the patient medical history, 
but it is rarely seen in our survey. One example appears in Star Trek: Voyager. 
In the episode "Riddles," we get a quick glance of one when the Doctor must 
treat the science officer, Tuvok (Figure 12.20). 

FIGURE 12.20a,b 

Star Trek: Voyager, "Riddles" (Season 6, Episode 6, 1999); Star Trek: Voyager, 

"Life Line" (Seasons 6, Episode 24, 2000). 


Chapter 12 


After diagnosing a problem, a physician decides on a course of treatment to 
heal the patient or, at minimum, alleviate suffering. Treatment interfaces 
are the easiest to depict in sci-fi because they generally involve physical 
devices or clear actions. In the survey, we see three main categories of 
treatment: devices for injecting medicine, some imaginative surgical 
interfaces, and a few systems for reviving the dead. 


Injections— whether of a sedative, vaccine, antidote, or painkiller— are 
important in sci-fi because the event is significant to the characters and to 
advancing the story. 

Perhaps the most famous injection device is Star Trek's hypospray 
(Figure 12.21). It allows medicine to be injected directly into the body 
without a needle puncturing the skin. Star Trek didn't invent these injectors. 
In 1947, a similar device was mentioned in an episode of the radio play The 
Shadow. And in 1960, Aaron Ismach patented such a device and won a Gold 
Medal from the US government for it in 1964, two years before the original 
Star Trek's premiere. But Star Trek's version did the real world even better— it 
could fit into the palm of your hand, it worked through clothing, and it didn't 
hurt at all. In a 1969 book called The Making of Star Trek, Gene Roddenberry 
explained that the hypospray was "invented" for the show as a means of 
circumventing NBC's prohibition against showing syringes piercing skin. 
Later Star Trek properties advanced the idea further, allowing doctors to 
select the medicine and dosage on the fly. 

FIGURE 12.21 
Star Trek: The Original 
Series, "Where No 
Man Has Gone Before" 
(second pilot, 1966). 



FIGURE 12.22a-e 
Sleeper (1973); Gattaca (1997); Men 
in Black 2 (2002); Total Recall (1990); 
Firefly, "Serenity" (Episode 1, 2002). 

Despite not having invented it, Star Trek popularized it with generations of 
fans, and it has become a staple of sci-fi medicine. The term hyposprayhas 
even leaked into other fiction properties, getting a mention just a few years 
later in the 1967 TV series Mission: Impossible. It has also leaked into the real 
world and is used interchangeably with jet injector in scientific papers. 

There are other injectors in sci-fi, but none seem to solve the problem as 
well. Other designs differ in three industrial design details: their component 
materials, how they are held, and whether the medicine inside is visible or 
not (Figure 12.22). 


Surgical treatments are a familiar staple in many genres. The bright lights 
surrounding an unconscious patient, the beeping machines, surgeons in 
scrubs shouting "Ten ccs, stat!" with sweat beading on their foreheads, and 
life most certainly hanging in the balance. Sci-fi is no exception. Surgical 
interfaces in the survey are common and show great variety. They have 
evolved over time from simple modesty panels to increasingly sophisticated, 
robotic, gestural, remote, and even volumetric projection technologies. 


Chapter 12 

In the Star Trek: The Original Series episode "Journey to Babel," Dr. McCoy 
performs a blood transfusion between Mr. Spock and his father, Sarek, 
while Sarek undergoes surgery. Sarek's torso is covered by a surgery canopy 
that gives McCoy access to Sarek but shields him from everything else- 
including Nurse Chapel (Figure 12.23). NBC had serious restrictions on what 
could be shown on prime-time broadcast television at the time, so this was 
likely a function of cultural modesty. But could there be a user-centered 
reason for this design? 

Perhaps the canopy prevents infection or assists the doctor in the mechanics 
of the surgery. It could have robotic arms applying and removing sutures 
or suctioning blood. It could house cameras so that remote doctors could 
observe and assist. It could even be a recording device for later study and 
witness in malpractice cases. The scene gives no clue about any of these 
purposes, however. It's left to our imagination. 

Another example is seen in Logans Run. The scene in the New You plastic 
surgery boutique fits into the film's overall social criticism that technology 
will worsen society's self-indulgent obsession with youthful appearance. At 
the boutique, any customer can drop in and select new facial features with a 
push-button interface (Figure 12.24). 

FIGURE 12.23 
Star Trek: The Original 
Series, "Journey to 
Babel" (Season 2, 
Episode 15, 1967). 

FIGURE 12.24 
Logan's Run (1976). 



FIGURE 12.25a,b 
Logan's Run (1976). 

The patient then lies down on a round, lighted table in the middle of a glass 
room, below what looks like a menacing, spiderlike, mechanical chandelier 
(Figure 12.25a). Each arm of the chandelier houses a laser and a nozzle that 
sprays a healing liquid. The system automatically calculates the necessary 
incisions and conducts them efficiently. In the scene, a physician overrides 
the safety protocols of the surgery machine in an attempt to kill the 
protagonist, Logan-5, apparently by setting it to "random" (Figure 12.25b). 

The physician's control interface seems entirely inadequate for the system 
to be operated manually with any kind of precision. The crude buttons, 
levers, and displays unconvincingly describe an interface capable of any 
sophisticated control. 

This kind of automated surgery is seen again in the movie The Island, 
in which clones are grown for organ harvesting. In one scene, a clone is 
placed in surgery to harvest his heart. The interface is impressively and 
appropriately complex. The physician uses a light pen on a radiographic view 
of the patient's body, alongside other controls, to specify the surgical work to 
be done. The surgery itself is done by robotic arms (Figure 12.26). 

A similar pen interface is seen a few years earlier in Minority Report, when 
John Anderton must have his eyes replaced to evade ubiquitous retinal 
identification systems. At an underground clinic, Dr. Eddie uses a desktop 
computer with a sophisticated real-time visualization of the eyes. Dr. Eddie 
specifies parameters by tapping a pen on the screen and then plans the 
actual surgery with a gestural interface, using lighted controllers worn over 
his fingertips (Figure 12.27). The surgery is then conducted by a complex 
actuator placed over Anderton's face. 


Chapter 12 


FIGURE 12.26a-c 
The Island (2005). 

In both Star Wars Episode V: The Empire Strikes Back and Starship Troopers, 
fallen warriors are brought back to health while suspended in vats of fluid. 
In the latter film, mechanical arms flutter back and forth weaving new tissue 
to repair Johnny Rico's leg wound. No interface to control treatment is seen 
in either film, though the transparent walls of the chambers are something 
of an interface for observation by caregivers and visits by well-wishers. 

In the film Chrysalis, Dr. Briigen conducts surgery standing in a surgical 
theatre where the translucent volumetric projection of her patient appears 
on a table before her (Figure 12.28a). She also sees representations of the 
small robotic arms doing the actual remote incisions. 

FIGURE 12.27a,b 
Minority Report 



V 1 A 

1 - ' * Jk ~Jmm 


FIGURE 12.28a-c 
Chrysalis (2007). 

Using a combination of gestures and voice, she is able not only to perform 
the surgical procedures but also to change scale and shift the perspective 
of the projection to see what she needs to see at the magnification that 
is most useful. She is able to achieve fine control using comfortably large 
gestures. In addition, she changes the projection so that only the relevant 
cardiovascular system is visible, eliminating from the display other data 
and imagery (Figure 12.28b). She is able to describe arbitrary planes of cross- 
section (Figure 12.28c). In the authors' opinion, this sequence is one of the 
most sophisticated, believable, and informed in our survey. It is a convincing 
prototype of such a system and, at the same time, works very well in the story. 



At the risk of fawning, the authors see this as a very well-real- 
ized prospective interface that should be seriously considered 
by designers working in this domain. It's easy to imagine that 
some technologies could get us partially there in the near term, 
such as 3D glasses in lieu of volumetric projection, without los- 
ing much of the brilliance of the design. 

Autonomous Medical Interfaces 

More and more frequently, technology is seen handling medical duties in 
sci-fi, either as robots or as artificial intelligences, such as the volumetrically 
projected doctor in Star Trek: Voyager. These generally provide the same 
services as human doctors, though the latter boast greater precision, less 
emotional detachment, and ready access to more medical information. 


Chapter 12 

As autonomous agents, they are designed to need little more than an 
interface capable of conversation and touch. 

Luke's medical doctor in Star Wars Episode V: The Empire Strikes Back is 
robotic (Figure 12.29). Its appearance is more humanoid, though with a 
severe, militaristic look. The robot is both surgeon and caretaker, with 
nimble mechanical arms. It's not clear if there's anything more than a 
pedestal below the robot's torso, and it doesn't vocalize even though 
Luke speaks to it. 

Contrast this robot with the one that attends Padme in Star Wars Episode 
III: Revenge of the Sith (see "Assisting Birth," below). This midwife robot is 
a remarkably different kind of medical robot that is clearly designed for 
caring, particularly in its appearance. It has very rounded shapes and edges, 
with only a hint of human characteristics— head, eyes, ample breast, and 
rounded belly. Its soft coloring and matte textures speak of comfort and 
safety. The face alone balances humanlike structure with non-humanlike 
features. It has four jewel-tone "eyes" of unequal size, with the location of the 
two larger ones mimicking the human face, and the two smaller ones in an 
odd asymmetric grouping. Still, its design suggests a gentle softness while 
maintaining a professional distance. 


Industrial designers have known this for a while, but the robot 
examples from Star Wars remind us that the surface design of 
medical technology can communicate different emotions and 
feelings to the patient. Warm nurturing and cool functional- 
ity may each have its place, but designers should take care to 
consider which is most important to their users. 

FIGURE 12.29 

Star Wars Episode V: The Empire Strikes Back (1980). 



Case Study: The Doctor 

The ultimate medical technology maybe the volumetrically projected doctor 
from Star Trek: Voyager (Figure 12.30). When the ship's human doctor is 
killed in the pilot episode of the series, the ship's emergency medical program 
is activated. It is a volumetric projection of a human doctor that acts as a 
humanlike interface to the ship's medical system. Though originally designed 
for emergency situations only, it becomes the ship's only functioning doctor. 
Given its human appearance, it curiously never adopts — and the crew never 
bestows upon it — a name; it is referred to only by its role. The Doctor's program 
is made autonomous and allowed to grow and learn beyond its original 
programming. It has fully human speech capabilities, and because of the 
ship's detailed force-field holodeck technology, it has a full physical presence 
as well, able to touch people and lift equipment. In these senses it is very 
human, able to address its patients in a way that is familiar and comfortable, 
although its bedside manner sometimes borders on irritating. 

Still, it is not human, and it has some very nonhuman abilities. Because 
it is connected to the ship's computer, it has an encyclopedic knowledge 
of medical precedent, procedures, and the medical histories of the crew. 
Because it can turn off the force field that is its "skin," it can become 
incorporeal and move through force fields that quarantine patients. Because 
it is a program, it can also be sent via compressed data signals to other 
locations across the galaxy and appear there, complete with memories 
and experience. If it weren't for Voyagers isolation, one would expect the 
Doctor to become adopted as the dominant medical paradigm across the 
Federation in short order. 

FIGURE 12.30 
Star Trek: Voyager 

282 Chapter 12 



Given that the Voyager Doctor's human shape and appearance 
are merely a convention adopted for patients, could it adopt 
other forms for different purposes? Could it become a caring 
midwife to assist with a birth? If it has access to a food replica- 
tor, can it become a wet nurse? Could it raise confidence by 
appearing as the patient's favorite doctor from back home? Or 
the greatest doctor ever? How about Florence Nightingale or 
Mother Theresa? If a real-world designer is working on a tele- 
medical interface or an avatar, such questions become relevant. 


Humans are social animals. We are hardwired to be good at 
dealing with other humans. The Voyager Doctor reminds us 
that even with a technology combining an infinite medical da- 
tabase, instant access to a patient's medical record, and an in- 
finitely malleable appearance, there is something comforting in 
wrapping these inhuman capabilities in a very human wrapper. 

Life and Death 

A few interfaces help with fundamental moments of medicine: assisting 
birth, revival, and signaling death. 

Assisting Birth 

There is very little birth technology in the survey. (Indeed, there may be 
very little in sci-fi. An Internet Movie Database search for "birth" in the plot 
summaries for the genre "Sci-Fi" only lists around 50 titles, and they are 
mostly obscure.) Scenes of birth in sci-fi largely focus on the human drama 
and don't support the tale with the medical interfaces surrounding it. The 
only example we've seen comes from Star Wars Episode III: Revenge of the 
Sith, as Padme gives birth to Luke and Leia (Figure 12.31). 

In this scene, we see volumetric displays with monitoring information. These 
seem to be out of view of the mother and only contain information about 
Leia; monitoring of the twins is not evident. There is no other doctor in the 
scene, so the presence and function of these displays is hard to explain (as is 
the width of the "modesty skirt"). The only other technology particular to the 
birth is the midwife robot. (See "Autonomous Medical Interfaces" above for 
more on this robot.) 

Medicine 283 

FIGURE 12.31 

Star Wars Episode III: Revenge of the S/th (2005). 



Even though many mothers choose natural childbirth, they may 
be interested in new technologies that display the health of 
the newborn and some sense of progress through the ordeal. 
Would a real-time visualization of the child through the birth 
canal help keep a mother focused on the physical effort she 
must undertake? Would confirmations that important stages 
had been passed without any complications be reassuring to 
her? Would an audible heartbeat help the doctor keep ambient 
track on the newborn's stress? Could some household or taxi- 
cab technology assist a woman through a birth that happens 
before she can get to a medical facility? 

There also seems to be an absence of prenatal technologies 
in sci-fi. With the glut of volumetric displays, what mother 
wouldn't want to see and reach out a hand to the image of the 
child growing inside of her? 


We see revival from death or the brink of death fairly often in sci-fi, partly 
due to the fact that it's easy and dramatic, and partly because it seems 
impressive, as this isn't something our current technology can do. There 
are no particular trends that stand out among these technologies and 
interfaces, and of course, they are all highly speculative. 

In The Day the Earth Stood Still, when the humanlike alien Klaatu is killed, 
his robot, Gort, carries his dead body into the spaceship and lays him on a 
special table. Gort waves a hand past some lights, and flips a switch on the 

284 Chapter 12 

FIGURE 12. 32a, b 

The Day the Earth Stood Still (1951). 

FIGURE 12.33 
Torchwood (2009). 

wall (Figure 12.32a). The cradle in which Klaatu's head rests glows with a 
bright light and he wakes up as if from a brief sleep. The only interface is the 
glowing cradle and transparent rod pointed at Klaatu's head. (Figure 12.32b) 
There is no diagnosis and no dispensing of medicine. Because the spaceship 
is so small, we can assume that this is a general-purpose medical table and 
not one with the sole purpose of reviving the dead. 

In the British TV series Torchwood, an alien glove called the "resurrection 
gauntlet" can temporarily revive the dead for a few seconds— enough 
time, perhaps, to ask the questions needed to determine a victim's killer. 
The gauntlet is worn on the hand and is simply placed against the dead 
person's head (Figure 12.33). The revived person is confused (they don't 
realize they're still dead), and much of the short time they're revived is spent 
explaining what's happening. Such objects are often found by the Torchwood 
operatives, so there are no instructions for the gauntlet, and the agents don't 
know how to use it well. Still, it showcases a direct, physical interaction and 
a simple, ergonomic industrial design. 



FIGURE 12.34a-c 
The Fifth Element 

A more extreme example of reviving someone occurs in The Fifth Element, 
when the only remaining part of the character is her hand. It's placed in a 
chamber in which the rest of her body is reconstructed in its entirety from 
her DNA (Figure 12.34). This system is completely automated and even 
includes an abort function in case what is produced is dangerous. 

Signaling Death 

If a character has been devoured by a ravenous Bugblatter beast, sci-fi 
makers don't need to provide any additional signals to the audience to let 
them know that death has occurred. Sometimes it's a great big text label, 
like we see in the Star Trek reboot (Figure 12.35). 

But a quieter, more dignified death requires a character like Dr. McCoy to 
say something like "He's dead, Jim," or some signal from a nearby technology. 
Fading lights are the simplest and most common signal. This is indicative 
of the association frequently seen in sci-fi that life is a light inside of things. 
When life ends, the light fades. 


Chapter 12 

FIGURE 12.35 
Star Trek (2009). 

A fading light interface appears in the dying biotechnical "bug" that 
Trinity extracts from Neo in The Matrix. After it is removed and tossed on 
the ground in the rain, its single red light slowly fades (Figure 12.36). The 
audience needs no other explanation of what has happened. 

FIGURE 12.36a-c 
The Matrix (1999). 

Medicine 287 

Men in Black had a subtle twist on this notion, as the little Arquillian in 
the mechanical human disguise dies at the same time as the lights fade. 
Had it been a failure of life support, we would expect to see the lights 
fade some time before the little guy passed away. But because it happens 
simultaneously with his death, either his body powers the robot many times 
his own size, or it describes a socially sensitive, context-aware interface that 
dims solely out of respect for the dead (Figure 12.37). 

FIGURE 12.37a,b 
Men in Black (1997). 


Chapter 12 

Another common way that the death of a character is signaled to the audience 
is one borrowed from medical dramas — waveforms in the monitoring system 
go flat, and alarms beep until a somber attendant turns them off. 

The design of monitoring visualizations determines how death is signaled. 
So as long as waveforms continue to appear, we can expect flat lines to be 
the harbinger of the worst news (Figure 12.38). 



Medical technology in the real world can often be insensitive 
to the emotions of patients, caretakers, and loved ones. What 
good is a persistent alarm long after the chance for revival has 
passed? Can medical technology take such grim circumstances 
into account and fade signals that could be distressing to the 
others present? 

h T1 — 

RECORD tt * *** * *** 

1 : 

-.r> jj c* 



■ I 


FIGURE 12.38a-d 

2007; A Space Odyssey (1968); Space: 1999 (1975); Brainstorm (1983); 

Aliens (1986). 



Sci-Fi Medical Interfaces Are Focused 
Mainly on the Critical Situation 

Technology evolves around us at an exponential rate. By comparison, our 
physiology's evolutionary rate is practically frozen. Even if we eventually zip 
around the galaxy at speeds faster than light, we'll be doing it with bodies 
that are similar to what we have now — and we're still going to get headaches. 
Medicine in sci-fi grounds the stories in that part of the future we can all 
understand — our bodies. 

Perhaps this is why some aspects of medical technology are completely 
unaddressed in the survey. Maybe sci-fi makers are more interested in the 
fantastical stuff that does change quickly. It's likely that this is what drew 
them to sci-fi in the first place. 

But sci-fi can't stretch things too far; it can only extend modern paradigms. 
With medical interfaces, we benefit from this forced nearer-term focus when 
good design thinking results in interfaces that propose new, feasible, and 
problem-solving technology that inspire real-world innovation. In this way, 
sci-fi medicine leads to the very real possibility of making life better here 
on Earth. 

290 Chapter 12 


Matchmaking 292 

Sex with Technology 295 

Coupling 300 

The Interface Is Not the Sex 307 

FIGURE 13.1 

Creation of the Humanoids (1962). 

Esme explains to her brother that her blue-skinned robotic servant and 
lover, Pax, is "dedicated to keeping me happy. And I am happy." With 
a hint of disgust, Cragis replies, "You love that . . . that machine?" 
Esme leans forward for emphasis to explain, "I love Pax" (Figure 13.1). 

Sex is a major part of the human experience, so it's no surprise that it plays 
a role in sci-fi. The sex-related interfaces seen in our survey fall into three 
primary categories. They can be distinguished by how the technology is 
related to the people or person having sex. 

• Matchmaking: technology helping people meet for sex 

• Sex with technology: people having sex with technology with no other 
human involved 

• Coupling: people having sex, with technology either enhancing or 
mediating the experience 


Matchmaking technologies help people meet for romance or sex. They either 
allow users to specify their desired aspects in a mate, or help people meet 
up with others interested in having sex. We found only four examples of 
matchmaking in our survey. 

In the film Logan s Run, Jessica-6 puts herself on the Circuit, a system that 
teleports people seeking sex from one residence to another until they find 
a partner. In practice, it works something like a cross between a radio and 
a Star Trek transporter. When Logan-5 wants to finish his evening with 
someone, he grabs a device that looks like a remote control, turns on 


Chapter 13 

FIGURE 13.2a-c 
Logan's Run (1976). 

ISO "?±: 

the Circuit, and spends a few seconds "tuning" the channel to solidify a 
candidate in the chamber. The first is a male, but this isn't what Logan is 
looking for (Figure 13.2a). He "detunes" this candidate and tunes in another 
one, which turns out to be Jessica. Liking what he sees, he extends his hand 
and helps her out of the chamber (Figure 13.2b, c). 

This early sex-related technology is difficult to evaluate as an interface 
because, although a device is used to turn on the Circuit and select 
candidates, we never see it in any detail. The most we see is Logan turning 
a dial at the top of a remote control-like object. Is he slowly toying with 
some variable? What would that setting be? Or are offerors broadcasting 
their bodies to different frequencies like radio stations on a dial, and Logan 
is just scanning channels? Offerors couldn't be at multiple places at once, 
so perhaps it is more like the video chat site Chatroulette, which connects 
people one-to-one but at random. Unlike Chatroulette, however, we see that 
Jessica-6 has no interface for controlling her end of the exchange, which 
raises questions about what the experience is like from her point of view. 

Additionally, because Logan first gets someone he's not interested in, there's 
either no ability to set preferences, or the preferences aren't specific enough, 
or Logan happens to be in a very particular mood and the preferences are 
difficult to change on the fly. An ideal filtering system should have prevented 
Jessica from materializing in Logan's apartment since he is a Sandman, a 
form of law enforcer, in which she has no interest. 

In Weird Science, two teenagers, Gary and Wyatt, specify the aspects of their 
ideal woman. To do this, they insert into a scanner clippings from magazines 
that represent their desired mental and physical traits (Figure 13.3a). The 
movie does not indicate how the system knows to imitate the legs of the 
Playboy model and the intelligence of Einstein, rather than vice versa, but 
the scene is a lighthearted montage, so many steps in the process are left to 
the audience's imagination. The boys are able to adjust some aspects of their 
ideal woman, such as breast size, with their keyboard (Figure 13.3b). 



FIGURE 13.3a,b 
Weird Science (1985). 

FIGURE 13.4a,b 
Total Recall (1990). 

In Total Recall, John specifies the type of love interest he'd like in his 
manufactured memory vacation by answering three multiple-choice 
questions about hair color, body type, and sexual aggressiveness. Any user 
of an online matchmaking system today knows that these feel woefully 
inadequate to specify an ideal love interest, but perhaps other variables are 
inferred through John's interaction in the implanted vacation (Figure 13.4). 

In the TV series Firefly, the character Inara is a companion, a kind of highly 
trained courtesan whose services to her clients often include sex. To select 
a client, she sends advance notice about when she will be in a location, and 
potential clients from that location send her videos to appeal to her to choose 
them. Her touch-screen interface allows her to review video applications, 
dismiss those she wants to reject, and make direct video contact with 
applicants (Figure 13.5). The system includes collaborative filtering by the 
network of companions so that dangerous clients are excluded. 

Why are there so few examples of matchmaking in the survey? All but Firefly 
appeared before real-world online dating sites became commonplace. Once 
much of the audience had direct experience with versions of these systems, 
such technology no longer seemed futuristic. Additionally, audiences now 
know that they have to supply quite a bit of information to get a good match, 
and this process is just not very cinemagenic. 


Chapter 13 

FIGURE 13.5 
Firefly, "Shindig" 
(Episode 4, 2002). 



Given some of the more recent developments in technology, 
there seems to be plenty of unexplored potential in match- 
making technology. For example, can preferences be respect- 
fully derived from social media streams and public datasets? 
Can systems help users move past unhealthy partner-seeking 
habits? Can the system prescreen people by evaluating their 
friends' experiences or ratings? Using ubiquitous sensors 
and subtle actuators, how "magic" and subtle could proximal 
matchmaking become? If a central computer knew that two 
people were made for each other, could it use ambient or 
augmented reality technology to help him notice her across 
a crowded bar by amplifying her laugh, or even by slightly 
brightening the light above him when she looked his way? 
Could matchmaking technology, like love, be "in the air"? 

Sex with Technology 

Another category of sex-related interfaces consists of people having sex with 
technology in some form. When such sex technology is physical, it can range 
in appearance from mechanistic devices to being nearly indistinguishable 
from sex with a real person. 


Sex devices are rare in the survey, with only two examples. Both are depicted 
as dystopian. In the first example, THX-1138, the oppressive state has provided 
technology to address and control citizens' basic needs for sexual release. 

At home after a hard day at work, THX-1138 sits down on a couch and turns 
on a volumetric projection of a woman dancing sensually to percussive 
music. A machine drops down from the ceiling, latches on to his penis, and 
mechanically moves up and down for exactly 30 seconds until he ejaculates. 
Then its tiny red light switches from red to green, the machine retracts 
back into the ceiling, and he begins flipping through channels to find other 
entertainment (Figure 13.6). 

Sex 295 

FIGURE 13.6a-c 
THX-1138 (1971). 

FIGURE 13.7a,b 
Sleeper (1973). 

The comedy Sleeper illustrates how transactional and meaningless sex 
has become in the future when Luna and a guest decide to "have sex" in a 
device called the Orgasmatron, which is about the size of a phone booth. To 
activate it, the two step inside and slide the door closed. A red light at the 
top illuminates, some moaning is heard, and six seconds later a green light 
indicates they're done (Figure 13.7). When they emerge, they continue their 
conversation as if nothing had interrupted it. The closing and opening of the 
door seem to be the only interface needed. 


The sex machines in both THX-1138 and Sleeper display a small 
red light while the device is working, and quietly switch to a green 
light when done. In Sleeper it's used for comedic effect and in 
THX-1138 to help describe a dystopia, but the message tele- 
graphed by the interface in each is the same. Sex, which for most 
is deeply engrossing experience, has been reduced to a small and 
disposable transaction. These interfaces would not speak to this 
disconnect if they had rich visuals and swelling music. Designers 


Chapter 13 

of real-world products and services should take care to avoid this 
same mismatch. The interface not only enables use but also in- 
forms the entire experience. Functional and cold may be right for 
some applications, but if yours is meant to be rich and engaging, 
the interface should embody that as well. 

There are a lot of analogous examples of sex interfaces in the real world — far 
more than we see represented in sci-fi. From low-tech masturbation sleeves 
like the Fleshlight and Fleshjack to more sophisticated devices like the Real 
Touch and sex machines, a wide variety of real-world products used for sexual 
gratification are absent from sci-fi, even when sex technology is depicted. 


Sexbots are androids capable of sexual intercourse with a human. Sexbots 
are by far the most common example of sexual technology in the survey 
(Figure 13.8). 

The reasons for this are many: they are easy to write for, they don't add to 
the special effects budget, and the sexual appeal of a sexbot does not need 
much explanation for the audience. In addition, sexbots create considerably 
less squeamish reactions in audiences than more mechanistic devices. It 
also means that the largest group of sex-related technology in the survey is 
not accessed through a visual interface but through a social interface: voice, 
gesture, and touch backed by some level of artificial intelligence. The one 
exception to this rule is the LoveBot married to Mr. Universe in Serenity, but 
his remote control for her is only seen in passing (see Figure 13. 8h). 

In the TV series Buffythe Vampire Slayer, the lovelorn vampire Spike has a 
sexbot created specifically to indulge his sexual and domination fantasies 
of Buffy (see Figure 13. 8f). In one scene, while they are engaged in foreplay, 
the Buffybot says to him, "Spike, I can't help myself. I love you." Spike, 
emboldened by the confession, replies, "You're mine, Buffy." After a pause the 
Buffybot asks, "Should I start this program over?" A troubled look crosses 
Spike's face and he says, "Shh. You're not a program. Don't use that word. 
Just be Buffy." 


Technology can be a good stand-in when the real thing isn't 
available, but the point of simulation is verisimilitude— allowing 
a person to suspend his or her disbelief. Exposing the techno- 
logical truth at the wrong time can draw attention back to the 
unavailability of the real thing, that the emotions may be ersatz, 
and seriously spoil the mood. With sexual technology in par- 
ticular but virtual reality in general, designers should be aware 
of and respect the natural ebbs and flows of social momentum, 
and avoid exposing the technology at inopportune moments. 

Sex 297 


FIGURE 13.8a-h 

Dr. Gold foot and the Bikini Machine (1965); Westworld (1973); Austin Powers: 

International Man of Mystery (1997); A.I. Artificial Intelligence (2001); 

The Stepford Wives (1975); Buffy the Vampire Slayer, "Intervention" 

(Season 5, Episode 18, 2001); Battlestar Galactica, "33" (Season 1, Episode 1, 

2004); Serenity (2005). 


Chapter 13 

Virtual Partners 

Sexbots are physical, but sex partners can be virtual as well. 

In the "Blood Fever" episode otStar Trek: Voyager, the Doctor prescribes a 
holodeck remedy to satisfy the ponfarr sexual needs of the Vulcan crew 
member named Vorik, because there is no female Vulcan within light years. 
In a similar plotline from the "Body and Soul" episode of the same series, the 
Vulcan, Tuvok, satisfies his ponfarr urges and avoids philandering by having 
sex with a holodeck version of his wife, who is on the far side of the galaxy 
(Figure 13.9). In Star Trek: Deep Space Nine, the Ferengi merchant Quark 
often rents out holosuites for sexual purposes. These virtual partners are for 
all practical purposes the same as sexbots because, to the users, there is no 
sensory difference. 

In The Matrix, Mouse salaciously assures Neo that he can arrange a more 
"intimate" experience between Neo and "the woman in the red dress," who 
is a character seen in a virtual reality training program (Figure 13.10). We 
never see this offer accepted or fulfilled, but because the virtual reality is 
indistinguishable from a real world, we can assume that such an encounter 
would work almost exactly like one with a sexbot or in the holodeck. 

FIGURE 13.9 
Star Trek: Voyager, 
"Body and Soul" 
(Season 7, Episode 7, 

FIGURE 13.10 
The Matrix (1999). 





When sci-fi sex technology is virtual— as in Star Trek's holo- 
deck or the Matrix-like virtual reality called the Construct— we 
see only virtual replacements for humans (or humanoid spe- 
cies). Given the infinitely malleable nature of these systems, 
a much greater range of sexual experiences and expressions 
are possible. Could someone choose new shapes, like a swan, 
or a centaur, or a robot? Exactly how do you want your furry 
avatar to look? 


Coupling technologies help humans have sex in some way. We have 
identified two distinct subcategories of coupling. The first uses technology 
to provide subtle clues that help set the mood, as in the case of augmented 
coupling. In the second, mediated coupling, people have sex with technology 
playing an intermediary function. 

Augmented Coupling 

In augmented coupling, technology enhances an otherwise purely biological 
act. Interestingly, this is the one example in the survey that comes from 

In Sexworld, a pornographic send-up of the sci-fi movie Westworld, Ralph is 
astonished when sexy music interrupts his conversation with his partner — 
who is a sexbot — in the bedroom in which he is being seduced (Figure 13.11). 
He asks her, "Where did the music come from? Did you do that?" to which 
she replies, "What's the difference? It's here. Hold me." 

FIGURE 13.11 

300 Chapter 13 

Ralph is distracted by the music because it appears out of nowhere and 
catches his attention. The notion that he's being watched or cued into action 
probably doesn't help his self-consciousness. 


When altering the mood of a space through computerized light 
and sound changes, make the changes slow so they do not draw 
attention to themselves. Sudden changes pull focus from the task 
at hand, distracting the user from the experience or their goal. 



There are many stimuli that a computer could control through 
sophisticated actuators to subtly shift the mood to one of 
sexual arousal: music, temperature, color, luminosity, and scent 
to name a few. How subtle could these controls get and still be 
effective? How many of these stimuli can a system control, and 
to what degree? In the far future, could a seducer have spaces 
that change, weather that shifts, or landscapes that adjust 
themselves to help accomplish his or her goal? 


A cyborg is a human whose body has undergone substantial mechanical 
augmentation. There is only one example of sexual interaction with a cyborg 
in the survey. 

In the comedy Space Truckers, Cindy is simultaneously excited and terrified 
at the sexual prosthetic sported by the cyborg villain Macanudo. To turn the 
prosthetic on, he pulls a ripcord like a starter for a lawn mower, and we hear 
a leaf-blower-like roar as a cold light emanates from his crotch to illuminate 
both their faces (Figure 13.12). Unfortunately for Macanudo, the device breaks 
down and he has to spend a few awkward minutes repairing it. Like many sex 
interfaces in sci-fi, this device is depicted as undesirable and inhuman. 

FIGURE 13.12a,b 
Space Truckers (1996). 



Mediated Coupling 

In mediated coupling, two human partners have sex using technology as 
the enabling media, often precluding any physical contact. This subcategory 
showed the most interesting and forward-looking examples, as well as some 
of the most disturbing. 

The Demolition Man sex helmets are described on page 149 as brain 
interfaces. The mildly telepathic helmets, invented as a high-tech 
prophylactic, provide lovers noncontact, sensual visions of each other (see 
Figure 7.36). This is the one example of consensual mediated coupling we've 
seen. The following examples from The Outer Limits and the Lawnmower Man 
are used more coercively. 

Lawnmower Man features a scene in which Jobe treats Marnie to a trippy 
introduction to cybersex. He straps her into a virtual reality suit suspended 
in a human-scale gyroscope. He then straps himself into a similar device. 
In the virtual world, Marnie is, at first, delighted at her new embodiment 
and the strange new ways they intertwine and combine (Figure 13.13). 
But Jobe is psychotic, and the scene changes from trippy and sensual to 
unwelcome domination. 

FIGURE 13.13a-c 
Lawnmower Man 


Chapter 13 


This practice is already well established in bondage and dis- 
cipline, sadism, and masochism communities, but it applies to 
immersive sexual technologies as well. Exploring the edges of 
sexuality can be fun, but when it crosses the line into some- 
thing unsafe, scary, or nonconsensual, people need a way to 
stop the experience and regain a feeling of control. This can 
take many forms, depending on the technology of the system. 
In voice-command systems, it can be an actual word or phrase. 
Ideally this safeword is something that is instantly recogniz- 
able, easy to remember, and something that wouldn't be said 
or done accidentally as a user participates in a scene. New 
safewords can be hard to remember in the heat of the moment, 
so standards have emerged. Sarah Smellie's 2011 poll indicated 
that the top safewords of her readers are safeword [sic], ba- 
nana, pineapple, and red (referencing a stop signal). 1 

This lesson can apply to nonsexual interfaces as well. Giving users a persistent, 
highly accessible control to return to an initial state provides a comforting 
means of recovery that encourages fearless exploration. Websites have long 
employed Home buttons that provides this reassurance. Apple's iPhone has a 
single hardware button on its face that serves the same purpose. 

Another unscrupulous use of mediated coupling appears in the TV 
series The Outer Limits. The episode "Skin Deep" shows "ugly" computer 
programmer Sid donning a handsome volumetric disguise (Figure 13.14). 
Both he and his roommate use their stolen identities to seduce others who 
are unaware of their true appearance. 


32133 J 



* ■ iV 

FIGURE 13.14 
The Outer Limits, "Skin 
Deep" (Season 6, 
Episode 3, 2000). 

1 Smellie, S. (2011). 2011 sex survey: Safewords. Retrieved from 

Sex 303 

FIGURE 13.15a,b 
Barbarella (1968). 

Durand-Durand's musical Excessive Machine was first introduced as a sonic 
interface on page 114, but its main purpose seems to be sexual torture. He 
immobilizes a naked Barbarella within it (Figure 13.15a), and as he plays 
its organ-like keyboard, unseen mechanisms stimulate her to the point of 
agonizing ecstasy (Figure 13.15b). 

In addition to being one of the earliest gestural interfaces in the survey, 
the sex technology in Flash Gordon is also one of the few wearable sexual 
devices. In the movie, Ming first uses his ring to enthrall Dale. Then, while 
she is under the ring's hypnotic influence, he uses gestures from afar to 
stimulate her and test her sexual response as a prerequisite for marriage. 
The interface appears quite intuitive, with Ming moving his hand over her 
distant silhouette, aligning the perspective to give the illusion that he is a 
giant caressing her small form (Figure 13.16). 

FIGURE 13.16a-c 
Flash Gordon (1980) 


Chapter 13 



People are good at using their hands for physical manipula- 
tions. In gestural interfaces, designers can leverage this facility 
for identifying or indicating an action on a particular object in 
their environment. For example, imagine a user specifying a re- 
cipient in a crowd for a text message by "tapping" them (in the 
air) in forced perspective. Such interactions would be concrete, 
understandable, and well within people's natural expertise. 
Systems would need to be able to interpolate a user's view and 
map his or her hand position onto it, and wearable augmented 
reality seems to be in an ideal position to do just that. 

Ming's interface could not work exactly like the film shows us. If the forced 
perspective is working correctly for the camera, it is not working for Ming. 
From his perspective, his hand would be off to the side, not caressing 
Dale. Additionally, forced perspective doesn't work if you have binocular 
vision and both eyes open. Try it yourself with an object near you to see 
that the part you're not focusing on appears double, making apparent edge 
contact impossible. The closer the object, the more mismatched the image 
from each eye. 

This gives us an opportunity for apologetics as described in Chapter 
1. Perhaps Ming's teledildonics system is so advanced that the forced 
perspective is unnecessary. If the system were smart enough, it could 
watch Ming's eyes to identify his victim, and map his gestures to her shape, 
regardless of where he held his hand or how accurately they matched her 
outline. In this conception of the interface, holding the ring in front of him is 
just a way for the technology to follow his gaze and gestures. 


It is difficult for users to trace precise shapes in the air, espe- 
cially for sustained periods. Relieve the pressure for precision 
by defining gestures that are quite distinct. This lets the system 
admit more imprecise input while maintaining confidence of 
the user's intent. 

The last two examples of mediated coupling involve systems that let users 
experience prerecorded sex with another human. 

Brainstorm follows the development and commercialization of a sensory 
recording and playback system. The head-mounted component is called 
"the Hat." After one of the lab assistants wears the Hat to create an illicit sex 
recording, the director of the lab takes it home and creates an endless loop, 
nearly overdosing on the sensory overload (Figure 13.17). 

Sex 305 

FIGURE 13.17a-c 
Brainstorm (1983). 

In Strange Days, Lenny is a peddler of amateur full-sensory recordings that 
often include sex. As in Brainstorm, people recording sexual experiences 
or playing them back must wear the device. The device is called a SQUID 
and functions much like Brainstorms technology, although Strange Days 
was created more than a decade later, so its technology is much smaller 
(Figure 13.18). 

In each case, the interfaces are minimal. Recording and playback are 
executed with a head-mounted device with little to no interface. Playback 
is controlled by standard media controls, based on either a reel-to-reel film 
or a compact disc paradigm. As the main examples of asynchronous sex in 
the survey, these two movies hint at the possibilities of one-to-many sexual 
expression and gender swapping. 

FIGURE 13.18a-c 
Strange Days (1995) 


Chapter 13 

The Interface Is Not the Sex 

Looking at all of the sex-related technology in the survey, the most 
interesting point that emerges is the lack of interfaces. There are some 
remote controls, parametric tools for matchmaking, a helmet or two, and 
some machines, but the content of sexual technology is always other people, 
as evidenced by the overwhelming representation of sexbots. This makes 
sense, as the sexual drive originally evolved to increase the likely success 
of animal reproduction. The human sex drive is understandably optimized 
toward a connection with other people. Interface designers need to keep 
this squarely in mind: with sex-related technology, the interface is only 
an inconvenient means to a pleasurable end, and it should be as usable, 
integrated, and discrete as possible, so the user can focus on the more 
important thing at hand. 

In addition, the subject of sex poses a difficulty not seen in the other topics 
in this book. Where we see a mutually influential interplay between design 
and sci-fi almost everywhere else in the survey, we see mostly the opposite 
effect with sex technologies and interfaces. In sci-fi, sex tends to divert 
the viewer's attention from the story, but with sex tech in real life, sex is 
the story (or, at least, the activity). This explains why sex in films and on 
television is often treated in shallow, titillating ways. Writers and directors 
are hoping not to derail the narrative but still excite viewers with something 
unexpected, funny, or surprising. Sex tech developers, on the other hand, are 
honestly trying to augment or transform sexual experience for themselves or 
others, so their explorations are more interesting and realistic. 

Sex 307 


What's Next? 

Using Sci-Fi 310 

More Than Sci-Fi 311 

And Sci-Fi to Come 313 

Getting to the end of this book has been a long, fun journey. We have 
tried to cover the most well-known, beloved, and influential movies 
and TV shows, from the earliest silent sci-fi to today's hyper-real, 
computer-generated epics with scenes dedicated to "interface porn," which 
put the plot on hold to let us indulge in an interface of excessive coolness. 

We've looked at sci-fi interfaces through a number of different lenses. We 
looked at familiar aspects such as physical controls, various inputs and 
outputs, the psychological aspect of anthropomorphism, cutting-edge 
technologies such as brain interfaces, and broad-reaching applications such 
as learning and communication interfaces. 

We've covered well over a hundred lessons, ranging from reminders to get 
the details right to high-level principles and opportunities for future work. 

We didn't even get to all of it. We had to omit some topics for space reasons: 
investigations of chemical interfaces, weapons, spacesuits, and spaceships, 
to name a few. Some shows and movies were more obscure, just outside of 
our domain, or we just didn't have time to get to them. 

But now that we're here, let's come out of hyperspace, look at the stars 
around us, and consider what it all might mean. 

Using Sci-Fi 

We've gleaned a lot of lessons from sci-fi, but the first one from Chapter 1 is 
arguably the most important. Yes, we enjoy sci-fi. (It's safe to say we love it.) 
But we can also use it to improve the practices of interface, interaction, and 
experience design. 

We're aware of the practical limits of this lesson. Sci-fi is incomplete as the sole 
textbook of an interaction design education. As much as it does tell us, it's not 
real enough to teach us everything. We should keep in mind that sci-fi creates 
interfaces as a by-product. Its focus is entertainment, and this affects the 
interfaces that come from it. Only occasionally does sci-fi concern itself with 
whether a speculative interface might be right for real-world users. 

All this said, we've seen repeatedly that if an interface works for an audience, 
there's something there that will work for users. Finding what that thing is 
and using it for inspiration in our own work is part of how we can use these 
speculative interfaces. 


Sci-fi interfaces help create a reality that is coherent and makes 
sense for audiences. In this way, audiences are a class of users, 
and the test of a speculative interface is the audience's ability 
to follow the narrative. Users of real systems follow a narrative 

310 Chapter 14 

of use that needs to be similarly coherent. This similarity makes 
it possible to learn from what we see on screen despite the 
different purposes of these experiences. The tricky part is to 
isolate what is useful only for the narrative but not for the real 
world, but this is where our experience as designers comes in. 

More Than Sci-Fi 

In choosing a nontraditional medium to explore interface design, we've 
found a process for analyzing "outsider" interfaces in a way that can inform 
our real-world work. Sci-fi isn't the only such domain, though it is the one 
that informed this particular investigation. Readers can certainly refer to 
the process we described in Chapter 1 and apply it to other genres or media. 
We're confident that the techniques used in this survey and our approach 
to analysis (including techniques such as apologetics) can prove useful 
in studies of other domains of speculative technology. We've focused on 
sci-fi with the most obvious and direct interfaces, but certainly many more 
remain to be explored. 

For example, superhero and spy genres almost always include techno- 
gadgetry, as with Mission: Impossible: Ghost Protocol's gestural heads-up 
display for passenger navigation (Figure 14.1a). The steampunk genre 
also includes speculative technology, despite components that speak to 
a different era, as in Sherlock Holmes (Figure 14.1b). Sci-fi comedies, such 
as Spaceballs, have an additional layer of humor to tease apart from the 
technology (Figure 14.1c), but anything with speculative technologies is 
likely to have interfaces worth examining. 

Other domains contain speculative technologies, such as video games and, 
for the brave, the deep pool of text-based sci-fi. 

FIGURE 14.1a-c 

Mission: Impossible: Ghost Protocol 
(2011); Sherlock Holmes (2009); 
Spaceballs (1987). 

. RADAR., 

What's Next? 


For that matter, we don't have to stick to the world of entertainment. We can 
look at the speculative fiction in industrial films and design investigations. 

For example, General Motors' 1956 Motorama traveling auto show was one 
of the first examples of a corporation creating speculative fiction to promote 
its brand and its view of the future. It included Frigidaire's wonderfully 
melodramatic industrial film Design for Dreaming discussed in Chapter 2 
(see page 21). This film didn't portray a real kitchen, but a well-imagined one. 
Surely there are lessons to uncover here, as well. 

A more recent example is Apple's Knowledge Navigator films from the 1980s, 
commissioned by Apple CEO John Sculley to illustrate a vision of technology 
20 years in the future, based on ideas from Apple's research and development 
labs (Figure 14.2a). Interfaces and technologies in the film included voice 
recognition and response, hypermedia, online media, online collaboration 
and video conferencing, agents, and more. This project set the standard for 
technology companies to create their own visions, as Sun did with Starfire 
seven years later (Figure 14.2b). 

The truth is that much real-world design is, like sci-fi, fictional. Even when 
interfaces are designed for actual products and services, they follow a 
process that is inherently speculative. As designers, we build personas of 
users that are built on real research, but they're still fictional characters. 
We create scenarios (stories) for them to inhabit. The scenarios that are 
developed are fiction, not unlike those in a film or TV series. We then 
prototype solutions (still fictional) in our attempt to craft solutions that will 
work for real users, and we iterate much like writers iterate screenplays. 

FIGURE 14.2a,b 

Apple's Knowledge Navigator (1987); Sun's Starfire (1994). 


Chapter 14 

FIGURE 14.3a-c 

Prototypes in the "Snow White" design language for the Apple II 

family of products, by Frog Design (1983-85). 

Prototypes, like the famous "Snow White" design language developed by 
Frog Design for Apple's hardware in the early 1980s, tell an influential story 
of possibility and opportunity (Figure 14.3). Although none of these specific 
prototypes found their way to market, they had an important influence on 
the development of the products and services that did. They were design 
fictions that were crucial to the design process. 

Let's face it, most of what we create never gets produced. If we're lucky, one 
of the iterations makes it past the development phase, to market, and into 
the hands of customers. Only then does some of this fiction become fact, 
and the real and imagined join together on the road to creating something 
wonderful and new. 

And Sci-Fi to Come 

New sci-fi stories, visions, and interfaces are being imagined and created 
as you read this. Some might contradict what we've found so far. The 
daydreamers in us kind of hope they do. Most, however, will likely build 
on what has come before. Undoubtedly these new stories will give us new 
examples to consider, new trends to follow, and new lessons to learn. But as 
long as we keep an inspired spirit, an analytical eye, and a questioner's mind, 
we can keep peering out of the portholes, learning lessons, and making it so. 

What's Next? 313 




Learning Lessons from Science Fiction 

Use science fiction 13 

Mechanical Controls 

Build on what users already know 19 

Tighten feedback loops 20 

Use mechanical controls when fine motor control is needed 24 

Don't get caught up in the new for its own sake 25 

Mix mechanical and other controls where appropriate 26 

Gestalt is important to users 27 


Visual Interfaces 

The visual design is a fundamental part of the interface 31 

Use all capital letters and a fixed-width typeface to evoke the look 
of early computer interfaces 33 

Otherwise, AVOID ALL CAPS 34 

Help experts display their mastery 35 

Sans serif is the typeface choice of the future 37 

Incorporate typographic principles from print 39 

Sci-fi glows 40 

Future screens are mostly blue 42 

Red means danger 44 

Gray makes interfaces look like an early-generation GUI 46 

To create a unique interface, avoid single, common colors and 
the glow effect 48 

Use color coding to help signal connections or categories 49 

Use nonrectangular-shaped screens to make them look advanced 50 

Use transparency to order important information while 
preserving context 54 


Avoid the confusion caused by too many overlapping, 
transparent layers 54 

Use familiar, real-world controls for quick understanding 55 

Opportunity: Explore alternative ways to group and access controls 57 

Help draw user attention through scale 59 

Three-dimensional data makes use of users' spatial memory 62 

Use motion to draw attention, cautiously 64 

Use motion to create meaning 64 

Creative combinations of even common stylistic choices create 
a unique appearance 73 


Volumetric Projection 

Differentiate the virtual 80 

Share the joke, after a beat 80 

VPs should conform to the Pepper's ghost style 81 

VP systems should interpret, not just report 84 

Position VPs eye to eye 85 

Opportunity: Scale avatars according to their importance 86 



A great demo can hide many flaws 96 

A gestural interface should understand intent 96 

Opportunity: Complete the set of gestures required 102 

Deviate cautiously from the gestural vocabulary 102 

Use gesture for simple, physical manipulations, and use language 
for abstractions 104 

Opportunity: Design a third-person gestural interface 107 

Choose a narrative point of view that makes sense 107 

316 Appendix: Collected Lessons and Opportunities 


Sonic Interfaces 

Assign one system sound per system event 111 

Convey ambient system state with ambient sounds 112 

Opportunity: Consider using spatial sound for nonspatial information 113 

Opportunity: Make music in the interface 115 

Put information in the channel it fits best 1 16 

Require multifactor authentication 118 

Account for variation in accents 1 18 

Reduce vocabulary to increase recognition 1 19 

Ignore words not part of the recognized vocabulary 120 

Choose command words that are representative of the action 121 

Make it easy for the interface to know it is being addressed 122 

Conversational interfaces should follow human social conventions 123 


Brain Interfaces 

Let the user relax the body for brain procedures 135 

Use lights and real-time results to show reading 142 

Virtual reality worlds should deviate from the real world with caution 145 

Bioengineered technology is creepy 148 

Opportunity: Visualize brain-reading interfaces 154 


Augmented Reality 

Augment the periphery of vision 162 

Place augmentations at the current depth of focus 162 

Consider alternative channels for information 163 

Information is empowering, but don't distract the user 165 

Augmented reality is personal 166 

In augmented reality, everything is special 168 

Appendix: Collected Lessons and Opportunities 317 

Provide opportunities for the user to intervene 169 

Simple augmentation is fast; fast augmentation is simple 170 

Opportunity: Show the gray area 170 

Opportunity: Let the reticle fire 174 

Opportunity: When a human must fire, let the eye target 174 



Consider animal representations for low-functioning systems 181 

Beware the uncanny valley 184 

Design either for absolute realism or stick to obvious representation 185 

Conversation casts the system in the role of a character 187 

Use paralinguistic sounds expressively to trigger anthropomorphism 188 

Achieve anthropomorphism through behavior 190 

Use an agent's social pressure for a social agenda 192 

Anthropomorphized interfaces are difficult to create successfully 193 

Opportunity: Offer several perspectives on interpretable information 195 

The more human the representation, the higher the expectations 
of human behavior 196 



Signal while recording 200 

Balance ease and control in activation 202 

Signaling change of state isn't enough 202 

Minimize the number of controls 204 

The human is sometimes the ideal interface 205 

Compose, then send 206 

The goal is to contact a person, not use an interface 207 

Opportunity: Find people by attributes 207 

Use sound for urgent attention 208 

Include a signal in a second channel for urgent attention 209 

318 Appendix: Collected Lessons and Opportunities 

Place a visual signal in the user's path 210 

Tap to receive a call 212 

Signal connections visually and disconnections audibly 212 

Handle emotional inputs 214 

In audio-only calls, show sound level visually 216 

Focus on the person, not the medium 218 

Translate both the vocal and anatomical act of speaking 219 

Opportunity: Let users alter their appearance, subtly 221 



Facilitate a teacher's input 229 

Align learning experiences with real ones closely 230 

Make learning a game 230 

Raise the stakes progressively as skills build 231 

Model tasks over time 232 

Engage learners in multiple mediums across multiple senses 235 

Design for the capabilities of the new medium 237 

Balance ease of use with error prevention 239 

Add meaning to information through organization 239 

Make the content relevant to the learner 241 

Provide a safe space to learn 247 

Provide expert guidance and suggestions 249 

Suggest the best option without being asked 252 

Rely on a user's recognition rather than recall 252 

Provide a sense of progress 253 

Help learners understand different perspectives 253 

Support emotional learning 254 

Opportunity: Provide intricate physical scaffolding 254 

Opportunity: Facilitate sensitization 254 

Opportunity: Enable learning in groups 255 

Appendix: Collected Lessons and Opportunities 319 



Opportunity: Design preventative interfaces 260 

One is special 261 

Use the waveform to indicate vital signs over time 263 

Being useful is more important than looking impressive 264 

Make manipulations physical 267 

Opportunity: Design the future of medical testing 272 

Opportunity: Design the future of medical diagnosis 274 

Opportunity: Build volumetric telesurgery 280 

It should/ee/ humane 281 

Opportunity: Let the doctor change shape to fit the task 283 

People need people 283 

Opportunity: Visualize the child for the mother in real time 284 

When the chance for revival has passed, respect death with silence 289 



Opportunity: Make matchmaking modern 295 

Small interfaces advertise small-value experiences 296 

Avoid reminding people of the simulation 297 

Opportunity: Don't dream it, be it 300 

Use subtle cues 301 

Opportunity: Augment everything 301 

Give users safewords 303 

Use forced perspective for real-world object selection 305 

"Satisfice" gestural input 305 


What's Next? 

If it works for an audience, some part will work for users 310 

320 Appendix: Collected Lessons and Opportunities 


Some of the lessons that emerged out of separate investigations ended up 
describing similar concepts. These lessons are called out below with the 
meta-lesson as its category. 

Use escape commands 

A gestural interface should understand intent 96 

Make it easy for the interface to know it is being addressed 122 

Break stylistic rules to stand out 

To create a unique interface, avoid single, common colors 
and the glow effect 48 

Creative combinations of even common stylistic choices create 
a unique appearance 73 

Be aware of passe styles 

Use all capital letters and a fixed-width typeface to evoke the look of early 
computer interfaces 33 

Gray makes interfaces look like an early-generation GUI 46 

Interpreters need to correct 

VP systems should interpret, not just report 84 

Translate both the vocal and anatomical act of speaking 219 

Fit information to channels 

Put information in the channel it fits best 116 
Consider alternative channels for information 163 
Include a signal in a second channel for urgent attention 209 
Signal connections visually and disconnections audibly 212 

The number of selected objects connotes 
a mode 

In augmented reality, everything is special 168 
One is special 261 

Appendix: Collected Lessons and Opportunities 321 

Manipulation fits hands 

Use gesture for simple, physical manipulations, and use language 
for abstractions 104 

Make manipulations physical 267 

People like technologies that are like 

Conversational interfaces should follow human social conventions 123 

The human is sometimes the ideal interface 205 

Handle emotional inputs 214 

Support emotional learning 254 

It should feel humane 281 

People need people 283 

Guide attention respectfully 

Help draw user attention through scale 59 

Use motion to draw attention, cautiously 64 

Use sound for urgent attention 208 

When the chance for revival has passed, respect death with silence 289 

Balance ease of use and error prevention 

Balance ease and control in activation 202 
Balance ease of use with error prevention 239 

322 Appendix: Collected Lessons and Opportunities 

CREDITS Paramount; 1.1b: Motorola; 1.2: Universum Film; 1.4: MGM; 1.5: 20th 
Century FOX; 1.6: White House Photograph Office (1968), #192584, http://; 1.7, 1.8: Dynamic Matrix Display, Xenotran LLC (2002). 

2.1: Image Entertainment; 2.2: Universum Film; 2.3: Universal; 2.4: MGM; 
2.5: Paramount; 2.6: George Pal Productions; 2.7: MPO Productions; 2.8: 
Paramount; 2.9: Nemo Science Center collection, photograph by Ryan 
Somma; 2.10, 2.11: Paramount; 2.12: Universal. 

3.1, 3.2: Universal; 3.3: 20th Century FOX; 3.4: MGM; 3.5: Warner Brothers; 
3.7a: Paramount; 3.7b: Columbia; 3.7c-e: Warner Brothers; 3.9a: MGM; 
3.9b: Carolco; 3.9c: 20th Century FOX; 3.9d: Columbia TriStar/Touchstone; 
3.9e: Columbia; 3.9f: Touchstone; 3.9g: 20th Century FOX; 3.9h: Pixar/ 
Disney; 3.9i: Dreamworks; 3.10: Dreamworks; 3.11a: Paramount; 3.11b, c: 
20th Century FOX; 3.13a: Dreamworks; 3.13b: Universal; 3.13c: MGM; 
3.13d: 20th Century FOX; 3.13e: Dreamworks; 3.13f: Touchstone; 3.13g: 
Paramount; 3.14a: Columbia; 3.14b: Warner Brothers; 3.14c: Paramount; 
3.14d: Touchstone; 3.14e: Paramount; 3.15a: Disney; 3.15b: Warner 
Brothers; 3.15c: Universal; 3.15d: 20th Century FOX; 3.16: Paramount; 
3.17a: Warner Brothers; 3.17b: 20th Century FOX; 3.17c: MGM; 3.17d: 
Touchstone; 3.18: Paramount; 3.19: Warner Brothers; 3.20: BBC; 3.21: 20th 
Century FOX; 3.22: ITC; 3.23: Paramount; 3.24a: 20th Century FOX; 3.24b: 
Columbia; 3.24c, d: 20th Century FOX; 3.24e: BBC; 3.25: London Film 
Productions; 3.26a: 20th Century FOX; 3.26b: Columbia; 3.26c: Touchstone; 
3.26d, e: 20th Century FOX; 3.27: Dreamworks; 3.28: Paramount; 3.29: 
TriStar; 3.30: Warner Brothers; 3.31a: 20th Century FOX; 3.31b: Pixar/ 
Disney; 3.32: ITC; 3.33: New Line; 3.34: Paramount; 3.35-3.37: Lionsgate; 
3.38a: TriStar/Columbia; 3.38b: MGM; 3.38c: Bandai Visual; 3.39: 
Paramount; 3.40: New Line; 3.41: Touchstone; 3.42: Chris Lee/Square/Sony; 
3.43: Universal; 3.44: Pixar/Disney; 3.45: MGM; 3.46-3.49: Paramount. 

4.1: MGM; 4.2a: MGM; 4.2b: Universal; 4.3: 20th Century FOX; 4.4a: Carolco; 
4.4b: New Line; 4.4c: Warner Brothers; 4.4d: Universal; 4.4e: Gaumont Video; 
4.4f: Paramount; 4.4g: Disney; 4.5a, b: 20th Century FOX; 4.5c: TriStar; 4.6a: 
Columbia TriStar/Touchstone; 4.6b, c: 20th Century FOX; 4.7a: Universal; 
4.7b: 20th Century FOX; 4.8: 20th Century FOX; 4.11-4.14: 20th Century FOX; 
4.15a: New Line; 4.15b: Warner Brothers; 4.15c: 20th Century FOX; 4.16a: 
New Line; 4.16b: 20th Century FOX; 4.16c: Gaumont Video; 4.17: 20th Century 
FOX; 4.18: Pixar/Disney; 4.19a: Warner Brothers; 4.19b: 20th Century FOX. 


5.1: 20th Century FOX; 5.2a: 20th Century FOX; 5.2b: Warner Brothers; 5.3: 
Paramount; 5.4: 20th Century FOX; 5.5: Paramount; 5.6, 5.7: 20th Century 
FOX; 5.8a: Gaumont Video; 5.8b: 20th Century FOX; 5.8c: New Line; 5.8d: 
Warner Brothers; 5.8e: Likely Story/This Is That; 5.9: TriStar/Columbia; 5.10: 
Paramount; 5.11: 20th Century FOX; 5.12: TriStar/Columbia; 5.13a: TriStar; 
5.13b: Gaumont Video; 5.14a: Pixar/Disney; 5.14b: Paramount; 5.15: 
Paramount; 5.16a: 20th Century FOX; 5.16b, c: Paramount; 5.17, 5.18: Likely 
Story/This Is That; 5.19: TriStar/Columbia. 

6.1, 6.2: 20th Century FOX; 6.3: Columbia; 6.4: Paramount; 6.5: New Line; 6.6: 
Warner Brothers; 6.7: Universal; 6.8: Warner Brothers; 6.9: Universum Film. 

7.1: Universal; 7.2: Warner Brothers; 7.3: 20th Century FOX; 7.4a: Universum 
Film; 7.4b: MGM; 7.4c: TriStar/Columbia; 7.4d: New Line; 7.4e: Disney; 
7.4f: Paramount; 7.4g: 20th Century FOX; 7.5a: MGM; 7.5b, c: Columbia; 
7.5d: 20th Century FOX; 7.5e: Paramount; 7.7: 20th Century FOX; 7.8: 
Paramount; 7.9: 20th Century FOX; 7.10a: Warner Brothers; 7.10b: Focus 
Features/Universal; 7.11: Warner Brothers; 7.12: 20th Century FOX; 7.13: 
Gaumont Video; 7.14: TriStar/Columbia; 7.15a: NBC; 7.15b: MGM; 7.16, 
7.17: Paramount; 7.18: Columbia; 7.19: Universal; 7.20: Paramount; 7.21: 
Focus Features/Universal; 7.22: Universum Film; 7.23: Disney; 7.24: 20th 
Century FOX; 7.25, 7.26: Warner Brothers; 7.27: MGM; 7.28, 7.29: 20th 
Century FOX; 7.30: Warner Brothers; 7.31: Paramount; 7.32, 7.33: Alliance 
Atlantis; 7.34: 20th Century FOX; 7.35: MGM; 7.36: Warner Brothers; 
7.37, 7.38: Paramount; 7.39: From Miyawaki, Y., et al. (2008). Visual image 
reconstruction from human brain activity using a combination of multiscale 
local image decoders. Neuron 60(5): 915-29; 7.40a: Mattel; 7.40b: Uncle 
Milton Industries; 7.41: Emotiv. 

8.1: 20th Century FOX; 8.2: Gaumont Video; 8.3a: 20th Century FOX; 
8.3b: New Line; 8.4: Paramount; 8.5a: Pixar/Disney; 8.5b: Paramount; 8.6: 
Paramount; 8.7: 20th Century FOX; 8.8: Orion Pictures; 8.9: 20th Century 
FOX; 8.10a, b: 20th Century FOX; 8.10c: Paramount; 8.11: Paramount; 8.12: 
20th Century FOX; 8.13: Universal; 8.14: Paramount; 8.15: Carolco; 8.16: 
Paramount; 8.17: TriStar; 8.18: Orion Pictures; 8.19: 20th Century FOX; 
8.20: Paramount; 8.21a: 20th Century FOX; 8.21b: Orion Pictures; 8.21c: 
Columbia TriStar/Touchstone; 8.21d: 20th Century FOX; 8.22: 20th Century 
FOX; 8.23: Carolco. 

9.1-9.3: Sony; 9.4a: Honda; 9.4b: iRobot; 9.5a: Universal; 9.5b: Warner 
Brothers; 9.6a: 20th Century FOX; 9.6b: Universum Film; 9.6c: 20th Century 

324 Credits 

FOX; 9.7a: 20th Century FOX; 9.7b: Paramount; 9.7c: 20th Century FOX; 9.7d, e: 
MGM; 9.8: Based on Mori, M. (1970). The uncanny valley. Energy 7(4): 33-35; 
9.9: Warner Brothers; 9.10: Universal; 9.11: VCI Entertainment; 9.12: Pixar/ 
Disney; 9.13: Paramount; 9.14: Warner Brothers; 9.15: Microsoft; 9.16: Apple. 

10.1: Universum Film; 10.2a: Fox Searchlight; 10.2b: New Line; 10.2c: 
20th Century FOX; 10.3a: 20th Century FOX; 10.3b: Pixar/Disney; 10.4a, b: 
MGM; 10.4c: Columbia TriStar/Touchstone; 10.5: Paramount; 10.6: MGM; 
10.7a: 20th Century FOX; 10.7b: Gaumont Video; 10.8a: Warner Brothers; 
10.8b: MGM; 10.8c: TriStar/Columbia; 10.9: 20th Century FOX; 10.10: MGM; 
10.11, 10.12: Universal; 10.13-10.15: Paramount; 10.16: 20th Century FOX; 
10.17: Paramount; 10.18: TriStar/Columbia; 10.19a: Universal; 10.19b: ITC; 
10.19c, d: 20th Century FOX; 10.20: 20th Century FOX; 10.21: Paramount; 
10.22: MGM; 10.23a: Universal; 10.23b: Paramount; 10.23c: Touchstone; 
10.24: TriStar/Columbia. 

11.1, 11.2: Paramount; 11.3, 11.4: Argos/Village Roadshow; 11.5: Warner 
Brothers; 11.6: Universal; 11.7: 20th Century FOX; 11.8: Universal; 11.9: 
Columbia TriStar/Touchstone; 11.10: Carolco; 11.11: Paramount; 11.12, 
11.13: 20th Century FOX; 11.14: Columbia TriStar/Touchstone; 11.15: 20th 
Century FOX; 11.16: Universal; 11.17: London Film Productions; 11.18: 
Universal; 11.19: Warner Brothers; 11.20: Columbia; 11.21: Touchstone; 
11.22: Apple; 11.23: Paramount; 11.24: Columbia TriStar/Touchstone; 
11.25-11.34: Paramount. 

12.1: Paramount; 12.2: Universal; 12.3: Paramount; 12.4: MGM; 12.5a: 
Dreamworks; 12.5b: 20th Century FOX; 12.5c: ITC; 12.5d: 20th Century 
FOX; 12.6: Paramount; 12.7: ITC; 12.8: 20th Century FOX; 12.9a: 20th 
Century FOX; 12.9b: Starz/Anchor Bay; 12.10a: ITC; 12.10b: Paramount; 
12.10c: 20th Century FOX; 12.11: Universal; 12.12: Paramount; 12.13: New 
Line; 12.14: Courtesy of Edgar Mueller; 12.15: 20th Century FOX; 12.16: 
Courtesy of the Radiology 3D and Quantitative Imaging Lab at Stanford 
University School of Medicine; 12.17: Columbia; 12.18-12.21: Paramount; 
12.22a: Rollins-Joffe Productions/MGM; 12.22b, c: Columbia; 12.22d: 
Carolco; 12.22e: 20th Century FOX; 12.23: Paramount; 12.24, 12.25: MGM; 
12.26: Dreamworks; 12.27: 20th Century FOX; 12.28: Gaumont Video; 
12.29: 20th Century FOX; 12.30: Paramount; 12.31, 12.32: 20th Century 
FOX; 12.33: BBC; 12.34: Columbia; 12.35: Paramount; 12.36: Warner 
Brothers; 12.37: Columbia; 12.38a: MGM; 12.38b: ITC; 12.38c: MGM; 
12.38d: 20th Century FOX. 

Credits 325 

13.1: Genie Productions/Dark Sky Films; 13.2: MGM; 13.3: Universal; 13.4: 
Carolco; 13.5: 20th Century FOX; 13.6: Warner Brothers; 13.7: Rollins- 
Joffe Productions/MGM; 13.8a, b: MGM; 13.8c: New Line; 13.8d: Warner 
Brothers; 13.8e: Paramount; 13.8f: 20th Century FOX; 13.8g, h: Universal; 
13.9: Paramount; 13.10: Warner Brothers; 13.11: Essex Video/Electric 
Hollywood; 13.12: Goldcrest Films; 13.13: New Line; 13.14: Alliance Atlantis; 
13.15: Paramount; 13.16: Universal; 13.17: MGM; 13.18: 20th Century FOX. 

14.1a: Paramount; 14.1b: Warner Brothers; 14.1c: MGM; 14.2a: Apple; 
14.2b: Sun; 14.3: Apple. 

326 Credits 



3D data, and user spatial memory, 62 

3D display, 11 

3D File System Navigator, 30 

3D file systems, representations in 

sci-fi, 60 
3D maps, 11-12 
3D visualizations, for medical 

monitoring, 267-270 
2V2D interface, 54 
2001: A Space Odyssey (1968), 9,45 

artificial intelligence, 21 

camera lens, 217 

communication, 201, 205 

HAL-9000 computer, 122,187 

presentations, 232 

signal for death, 289 

telephone number entry system, 206 

waveform display in medical 
monitoring, 262 

abstract gestural control, 1 03 
abstract learning, 241 
accents in voice identification, 118 
activating communication system, 202 
active subjects, for brain interfaces, 

The Adjustment Bureau (201 1), 1 3 7 
Adobe Dreamweaver, code interface, 35 
Advanced Technology Group (Apple), 1 94 
Aesthedes computer, 22, 23, 56 
agency, degrees of, 190-195 
agency trigger, 190 
aggregated tag cloud, 8 
A .1. Artificial Intelligence (2001), 298 
AkzoNobel, 43 

alert word, computer listening for, 1 2 3 
Alien (1979), 56 

command-line interface on, 32, 33 

medical scanning, 266 

Mother (artificial intelligence), 186 

reticle design, 172 

robots, 182-183 

typefaces in, 36 

warning, 116 
Aliens [19&6), 92-93 

augmented reality, 172 

communication, 203, 204, 213 

medical monitoring, 263 

signal for death, 289 

stored communication contacts, 

terrain modeling, 164 
alphabetic organization, 240 
ambient sound, 112 
Americans with Disability Act, 54 
animation of interface, 62-64 
anthropomorphism, 178-196 

alien species and, 181 

appearance, 185 

audible expressiveness, 188 

behavior, 189-195 

care in invoking, 195-196 

categories, 182 

expectations for, 196 

humanness transfer to nonhuman 
systems, 179-185 

interface creation difficulties, 193 

voice, 186-188 
apologetics, design lessons from, 
9-10, 311 

Firefly heads-up display, 1 6 6 

Flash Gordon forced perspective, 305 

Lost in Space robot, 106 

medical volumetric projection, 

Star Trek language translation, 219 

Star Wars gunner seats, 112,162 

Knowledge Navigator (1987), 192,194, 

Project 2000, 240-241 

Siri, 119, 121, 207 
arch, in Star Trek holodeck, 93 
Arial typeface, 37 
artificial intelligence, 122 

absorption of human, 150 

HAL-9000, 122, 187, 217 



JARVIS, 165 

Knight Rider, K.I.T.T., 1 8 6 

piloting spaceship, 150-151 

voice for, 186-188 
ASCII encoding, 32 
ASIMO robot (Honda), 180 
assistance, for anthropomorphic 

systems, 191 
assistive medical interfaces, 259-280 

3D visualizations, 267-270 

diagnosis, 272-274 

evaluation, 260-272 

prevention, 259-260 

scanning, 266-267 

surgery, 276-280 

suspended animation, 265 

testing, 271-272 

treatment, 275-280 
astronaut, 18 
asynchronous communications 

vs. synchronous, 199-202 

volumetric projection for, 81 
attributes, finding people by, 207 
audible expressiveness, in 

anthropomorphism, 188 
audience expectations, 6. See also user 

audio communication, 214-217 
audiovisualization, 216 
auditory indicator, in medical 

monitoring, 261 
augmented coupling, 300-301 
augmented reality (AR), 158-176 

appearance of, 160 

awareness of people, 167-171 

depth of focus for, 1 62 

future of, 1 76 

goal awareness, 171-175 

holodeckand, 254 

interaction with, 176 

location awareness, 163-1 64 

for medical scans, 269 

object awareness, 165-167 

personal nature of, 166 

sensor display, 160-163 

speed and simplicity, 170 

technologies included, 158-160 
Austin Powers: International Man of 

Mystery {1991), 298 
authentication, multifactor, 1 1 8 
authorial intent, 10 
automobile technology, hands-free, 119 
autonomous agents, 192 
autonomous medical interfaces, 

autonomy trigger, 190 
avatar, 105 

cosmetic alteration, 221 

disguise for, 220 
Avatar (2009), 51, 52 

brain interface, 133 

glow in, 40 

message recording, 199-200 

re-embodiment technology, 94 

telepresence, 148 

volumetric projection, 86, 88, 89 
awareness of people, in augmented 
reality, 167-171 


Back to the Future Part II (1989) 

augmented reality, 167 

communication, 209-210 

volumetric projection, 79, 80 
Barbarella (1968) 

communication, 216 

music as weapon, 114,115 

sex technologies, 304 
Battlestar Galactica, 44 

color in, 42 

preventive medicine, 259 

robotic daggits, 181 

sexbot, 298 
behavior, in anthropomorphism, 

degrees of agency, 190-195 
beveled edges, on software objects, 54 
bioengineered technology, 1 47-1 4 8 
biological systems, 258 
biological understanding of life, 258 



biometric indicators, 260 
biophilia, 148 
birth technology, 283-284 
Blade Runner (1982), 45 

communication interface, 205 

conversational interface, 121 

reticle design, 172 

typefaces in, 36 

voice interface, 118-119 
blue, in sci-fi interfaces, 42,43 
bottom up, learning lessons from, 7, 1 

direct knowledge transfer, 225 

importance, 126 

physically accessing, 126-131 
brain/consciousness interfaces, 57 

active subjects, 144,144-151 

disabling the mind, 131 

dismantling myths, 151-153 

information directions, 132-144 

invasive, 126-127 

myths, 155 

noninvasive, 127-131 

playing game, 151 

reading from brain, 138-142 

real-world, 130 

remote connection, 130-131 

telexperience, 142-144 

user expectations for physical 
location, 130 

worn devices for, 128-129 

writing to brain, 132-138 
Brainstorm (1983), 33,143 

brain interface, 128 

communication, 201 

mediated coupling, 305-306 

signal for death, 289 
Brazil (1985), 26, 27 
Buck Rogers (1939) 

audio communication, 215 

brain interface, 126,128,131 

mechanical controls, 18,20 
Buck Rogers in the 25th Century (1979), 21 
Buffy the Vampire Slayer, 297,298 
buttons. See also mechanical controls 

C-3PO, 190 

Caldwell, Douglas, 11 

calls in communication process, 81 , 199 

accepting, 211-212 

ending, 213-214 

receiving, 208-214 
cameras, 217, 218 
capital letters in text, 33,34 
cathode ray tubes (CRTs), 32 
cause and effect, 17 
CDC (Control Data Corporation), 32 
The Cell (2000), 57 
change of state signal, 202 
character arc, learning and, 224 
Cheskin, 43 
Chicago typeface, 37 
The Chronicles ofRiddick (2004), 66-67 
Chrysalis (2007) 

augmented reality, 158-159 

communications, 204 

gestural interface, 97,100 

medical treatment, 279-280 

mind-writing device, 134 

second-person gestural interface, 104 

volumetric projection, 78, 87 
Chuck, 135 
Claessens, 22 
classroom environment. See also learning 

presentations, 236 
Clippy (Microsoft), 192-193 
Close Encounters of the Third Kind (1977), 

music interface, 114 
Coleran, Mark, 42 

for brain interface, 136 

for controls, 56 

for GUI, 41-49 

in Hitchhiker's Guide to the Galaxy, 65 

in The Incr edibles, 67 

for LCARS interface, 69 

original uses of, 46-48 

for volumetric pr oj ection, 7 9 
color coding, 48 



color schemes, 73 

for code interface, 35 
combinations, creativity for unique 

appearance, 73 
command-line interfaces (CLIs), 

as expertise indicator, 34 

vs. WIMP interfaces for reference, 239 

activating system, 202 

asynchronous vs. synchronous, 

audio, 214-217 

audiovisualization, 216 

basics, 198 

in Buck Rogers, 20 

composing message, 199-200 

disguise in technologies, 220-221 

ending call, 213-214 

future possibilities, 221 

goal of contacting person, 207 

language translation, 218-220 

message playback, 201 

monitoring connection, 212 

operator for connection, 204-205 

receiving call, 208-214 

specifying recipient, 203-207 

stored contacts, 206-207 

video, 217-218 

visual signal placement, 210 

volumetric projection for, 81-85 
composing message, 199-200 
computer interfaces 

nonrectangular screens, 50-51 

sounds from, 112 

typeface to indicate early, 33 
confusion, from volumetric 

projection, 87 
consistency, 5 
consumer brain- computer 

interfaces, 154-155 

abstract gestural, 103 

ease-of-use vs., 23 

by thought, 144 

controls (buttons, knobs, etc.) 

grouped, 55-58 

minimizing number, 204 

number of, 23 

protecting against accidental 
activation, 202 

screen-based, 25 
conversational interfaces, 116, 121-124, 

first, 110 

vs. limited voice-command, 1 2 
cost, as design constraint, 23 
coupling technologies, 300-306 

cyborgs, 301 

mediated, 302-306 
Courier typeface, 37 
Cowboy Bebop (2001), 60,61 
Creation of the Humanoids (1962), 292 
cropping, of volumetric projection, 88 
crosshairs on weapon (reticle), 172, 174 
cyborgs, 301 

danger, red as indicator, 44 
Dark Star -(1974), 187 
database of movies and TV 

properties, 7-10 
The Day the Earth Stood Still (1951), 21 

gestural controls, 92 

Gort, the robot, 182 

message composition, 199 

revival from death, 284-285 

sonic interface, 110 

respecting with silence, 289 

revival from, 284-289 

signal for, 286-289 
DEC (Digital Equipment 

Corporation), 32 
Defying Gravity (2009) 

glow in, 40 

medical monitoring, 263 
degrees of agency, 190-195 
deleting memories, 1 3 7-1 3 8 



Demolition Man (1993) 

mediated coupling, 302 

virtual sex, 149 
demos, 96 

depth of focus, for augmented reality, 1 62 
descriptions in database, attributes 

tagged in, 7 
Design for Dreaming (1956), 21, 312 
design lessons 

ambient system state, sounds for, 1 1 2 

anthropomorphism, 188, 190, 

from apologetics, 9-10 

appearance, 50, 73 

augmented reality, 162,166,170 

avoid simulation reminder, 297 

best option suggestion, 252 

bioengineered technology, 148 

color coding, 49 

communication, 123, 202, 207, 

controls, 204 

differentiating the virtual, 8 

emotional inputs, 214 

emotional learning, 254 

expert guidance, 249-250 

feedback loops, 20 

finding, 7-10 

flaws hidden by demos, 9 6 

games, 231 

Gestalt importance to users, 27 

gestures, 96, 104 

gray for interfaces, 46 

help experts display of mastery, 35 

humans, 205, 283 

ignoring words not in vocabulary, 1 2 

information, 116, 163, 165 

interface knowledge of being 
addressed, 122 

learning, 229, 230, 235, 241, 
247, 253 

mechanical controls, 24,26 

medical technology, 261-262, 
264, 281 

modeling tasks over time, 232 

motion, 64 

multifactor authentication, 1 1 8 

narrative point of view choice, 1 07 

new medium capabilities, 25,237 

organizing for meaning, 239 -24 

physical manipulations, 267 

realism vs. representation, 185 

real-world controls, 55 

real-world object selection, 305 

recognition vs. recall, 252 

reference ease of use vs. error 
prevention, 239 

respecting death with silence, 289 

resting position for brain writing, 1 3 5 

sci-fi glow, 40 

sexual technologies, 303 

shape of, 1 

signals, 44, 111, 142, 200, 208, 209 

similarities, 9 

social pressure of agent, 1 92 

spatial memory, 62 

subtle cues for sexual mood, 301 

telephone number entry system, 206 

text interfaces, 33, 34, 37, 39 

transparency, 54 

uncanny valley, 184 

unique interface creation, 48 

user attention, 59 

user intervention opportunities, 169 

user knowledge as base, 1 9 

virtual reality world deviation from real 
world, 145-146 

visual design in interface, 31 

vocabulary reduction to increase 
recognition, 119 

voice, 121, 187 

voice identification, 118 

volumetric projection, 80, 84, 85 
Destination Moon (1950), 20 
devices for sexual gratification, 

diagnosis, 272-274 
direct download for learning, 225-227 
directional sound, 112-114 
direct knowledge transfer to brain, 225 



direct manipulation, 102-104 
disguise in communication 
technologies, 220-221 
display shape, 50-51 
District 9 (2009), 53 

exosuit, 170 

gestural interface, 100 

volumetric projection, 79 
Doctor Who, "Rose," 50 
Dollhouse, 51, 52 

brain interface, 127, 129, 131, 133 

brain interface trigger, 132 

memory wiping, 138 

reading from brain, 140 

remote brain interface 
connection, 130-131 
DOS prompt, 1 7. See also command-line 

dreams, recording, 141 
Dreamweaver (Adobe), code 

interface, 35 
Dr. Goldfoot and the Bikini Machine 

(1965), 298 
drop shadow, 55 
Dune (1984) 

reference tools, 236-237 

scanning interface, 266 

voice interface, 120 

weirding module, 2 2 7-2 2 8 
dystopia, 27 

Eagle Eye [2W$), 38, 52 
ease-of-use, vs. control, 23 
Ebay auction system, agency vs. 

autonomy, 190 
Einthoven, Willem, 262 
electrocardiograph, 262 
Eliza (computer program), 1 9 
emoticon display, 178 
emotional inputs, 214 
emotional learning, 254 
emotive sound, 188 
EmotivEPOC, 155 

ending call, 213-214 

ENIAC, 32 

Enterprise (Star Trek prequel) 

interface, 45, 69, 70 

interface compared to Star Trek: Deep 
Space Nine, 46 
essence of living things, 258 
Eternal Sunshine of the Spotless Mind 
(2004), 132 

memory wiping, 138 
Eurostile typeface, 37, 67 
evaluation in medicine, 260-272 
eXistenZ (1999), 146-147 
exosuit, 92, 170 
expectations. See also user expectations 

of audience, 6 
expert users, interface for, 23 
eye contact in volumetric projection, 

eyestrain, 162 

from volumetric projection, 88 

failure, fear of, 247 

Fantastic Four (2005), 42 

Fantastic Planet (1973), 226 

fear of failure, 247 

FedPaint tool, 243 

feedback loops, tightening, 2 

fictional technologies, interface for, 3 

The Fifth Element (1997) 

reference tools interface, 238 

revival from death, 286 

screen interfaces, 43 

transparent displays, 51, 52 

typefaces, 38 
file management systems, 58-62 
The Final Cut (2004), 59, 60 
Final Fantasy (2001), 6 6 

design lessons, 7-10 

people by attributes, 207 
fine motor control, mechanical controls 
for, 24 



Firefly, 44,103 

audio communication, 215 

augmented reality, 159 

gestural interface, 97, 99 

matchmaking, 294 

medical mystery, 271 

medical scans, 269-270 

medical treatment, 276 

object awareness, 166 

reticle design, 172,173 

screen shape, 50 

volumetric projection, 79, 80, 87, 88 
fixed connection, for 

communications, 203-204 
fixed-width typeface, 33 
Flash Gordon, 64 
Flash Gordon (1980), sex 

technologies, 304 
flaws hidden by demos, 9 6 
Flight of the Navigator (1986), 4 4 

brain interface, 128 

language ability, 122 

reading from brain, 140 
floating projections, 83 
flying, as augmented reality 

awareness, 171-172 
Fogg, B. J., 179 
Foley, 111 
Forbidden Planet (1956) 

brain interface, 129 

communication devices, 203 

manifesting thought, 1 4 9 

robot, 182-183 

starship interface, 18,19 

thought interface, 154 

volumetric projection, 76, 78 
Force Trainer (Uncle Milton 

Industries), 155 
friend-or-foe assessment, 168-169 

gray area in, 170 
Frigidaire, 312 

Frog Design, "Snow White" design 
language, 313 

functional magnetic resonance imaging 

(fMRI) neuron reading, 152 
Futurama, 122 
Futura typeface, 37 

Galaxy Quest {1999) 

color in, 42 

typefaces in, 36 

warning countdown, 116 
Gamer (2009), 60, 61 

gestural interfaces, 108 

learning as, 230 

playing, 151 

raising stakes as skills increase, 231 
Gattaca (1997) 

DNA-testing device, 271 

medical treatment, 276 

typefaces in, 36 
gaze-matching problem, in volumetric 

projection, 82-83, 217 
General Motors, Motorama auto show 

(1956), 21, 312 
GERTY, 178-179 
gestural interface, 21, 25, 28, 92-108 

continuing development, 1 8 

continuing maturation, 97-98 

direct manipulation and, 1 02-1 04 

intent and, 96 

Minority Report example, 95-97 

narrative point of view, 1 04-1 07 

risks from changing vocabulary, 1 02 

terminologies included, 92-94 

third-person, 107 
gestural vocabulary, 98 

extend hand to shoot, 1 01 

pinch and spread for scaling, 1 01 

point or touch to select, 100 

push to move, 99 

swipe to dismiss, 100 

turn objects to rotate, 99 

wave to activate, 98 
Gilliam, Terry, 27 



glow, 40-41 

and transparency, 54 
goal awareness, in augmented 

reality, 171-175 
Google search, Minority Report vs. Star 

Trek interface, 95 
graphical language, for Enterprise, 6 9 
graphical user interfaces (GUIs), 17, 30, 

2V2D, 54 

color, 41-49 

display shape, 50-51 

file management systems, 58-62 

glow, 40-41 

grouped controls, 55-58 

layers and transparency, 51-54 

motion graphics, 62-64 

typography, 36-39 
gray, in interfaces, 46 
green, in interfaces, 44 
Gricean maxims, 123 
grouped controls, 55-58 
groups, learning in, 255 
guides, 192 

anthropomorphic systems as, 191 
Guides 3.0 (Apple), 192,194 


Hackers (1995), 60 

HAL-9000 computer, 122, 187, 217 

hands-free automobile technology, 1 1 9 

heads-up displays, 51 

Heinlein, Robert, 20 

help experts display of mastery, 35 

Helvetica typeface, 37 

The Hitchhiker's Guide to the Galaxy (2005) 

color in, 42, 43 

communication, 218, 219 

language ability, 122 

reference tool, 240 

screen interfaces, 43 

visual style, 65 

voice interface, 122 

holodeckinStarTreA;, 247-255 
features missing, 254 
unique capabilities, 253-254 
holograms, 76. See also volumetric 

Honda, ASIMO robot, 180 
humanness transfer to nonhuman 
systems, 179-185. See a/so 
AI absorption of, 150 
augmented reality awareness of, 

as ideal interface, 205 
need for others, 283 
revulsion for facsimiles, 183 
social conventions for conversational 

interfaces, 123 
social norms of, 180 
hybrid controls, 26 
hypotheses testing, 252 


IBM PC XT, 23 
icons, 36 

for file content, 60, 36 
images, to portray file content, 60 
Immelmann turn, 107 
The Tncr edibles (2004) 

communication, 201 

gestural interface, 101 

sensor display, 161 

typefaces, 38 

visual style, 67-68 
Independence Day (1996) 

augmented reality, 172 

typefaces in, 38 
industrial age users, mechanical 

feedback for, 17 
industrial design, 3 
industrial films, speculative fiction in, 

influence, reciprocal, 2 




direction in brain interface, 132-144 

empowering vs. distracting, 165 

extracting from brain, 138-142 
information architecture, 237 
information design, 3 
injections for medical treatment, 275 -276 
inspiration in science fiction, 11-13, 57 
intent, gestural interface and, 96 
interaction design, 3 

creating unique, 48 

denned, 3 

design, 2 

for sex, lack of, 307 

layers, 51-54 

real-world expectations, 2 

user familiarity with, 1 9 
internalization of knowledge, 241 
Internet Movie Database, 5 
interpretable information, perspective 

options for, 195 
intervention by users, opportunities for, 

invasive brain interfaces, 126-127 
iPad, 102 

/, Robot (2004), 182-183 
iRobot, Roomba vacuum robot, 1 8 
Iron Man (2008), 58 

augmented reality, 160, 171 

biographical information display, 1 67 

color in, 42 

communication, 209-210, 211, 212, 

Dummy (robot), 189 

gestural interface, 103 

object awareness, 165 

sensor display, 161 

terrain modeling, 164 

threat assessment, 169 
Iron Man 2 (2010) 

gestural interface, 99,101 

screen interfaces, 43 

touch-screen gestures, 94 

volumetric projection, 78 

The Island (2005) 

automated surgery, 278 

color in, 42 

medical monitoring, 263 

text display, 38-39 
Ismach, Aaron, 275 

JARVIS (artificial intelligence), 1 65 
Johnny Mnemonic (1995) 

brain interface, 128,134 

communication, 205, 214, 220 

file management, 60,61 

gestural interface, 98, 100, 105-106 
joke with volumetric projection, 8 
Jones, Katherine, 57 
Jurassic Park (1993), 30-31,57 


Kinect (Microsoft), 108 
Knight Rider, K.I.T.T. (artificial 

intelligence), 186, 190 
knowledge. See also learning 
installing and uninstalling in 

brain, 153 
internalization of, 241 
Kobayashi-Maru test in Star Trek, 

Lang, Fritz, 182 

ability, 122 

for abstractions, 104 

responsive use of, 186 

translation, 218-220 
The last Starfighter (1984), 11, 218, 219 

psychomotor practice, 229 
Lawnmower Man (1992), 128,302 
layers in GUIs, 51-54 
LCARS (Library Computer Access and 
Retrieval System), 22, 48, 62, 

frame graphics in, 55 




abstract, 241 

aligning experiences with reality, 230 

character arc and, 224 

content relevance, 241 

emotional, 254 

in groups, 255 

holodeck, 247-255 

providing sense of progress, 253 

safe space for, 247 
learning interfaces 

categories, 224 

direct download, 225-227 

future, 255 

machines for thinking, 241 -244 

presentation tools, 232-236 

psychomotor practice, 227-231 

reference tools, 236-241 

testing, 244-247 
LED-type faces, 37 
lessons, downloaded to the mind, 

Lifted(2006), 56, 81, 89 
lights, to indicate brain reading, 1 42 
limited command voice interfaces, 

vs. conversational, 120 
living things, essence of, 258 
locational data, organizing, 31 
Logan's Run (1976), 277-278 

communication, 208 

images, 76 

interface, 32, 68 

matchmaking, 292-293 

typefaces, 38 
Lost in Space (1998), 63 

augmented reality, 159 

gestural interface, 97, 101, 104, 106 

medical monitoring, 267-268 

medical scans, 270 

message recording, 199-200 

voice security, 117 

volumetric projection, 78, 86, 87, 88 

lowercase characters, in text-based 

interface, 32 
Luxojr. (1986), 188 


machines for thinking, 241 -244 

holodeck as, 251-252 
The Making of Star Trek (Roddenberry), 

The Manchurian Candidate, 1 3 2 
manipulation, direct, 102-104 
masslessness, 88 
matchmaking, 292-295 
The Matrix (1999) 

brain interface, 127,133 

death in, 287 

knowledge-download interface, 227 

program represented as human, 185 

typefaces in, 36 

virtual partners, 299 

virtual telepresence, 144-145 
The Matrix Reloaded (2003) 

color use, 44, 46, 47 

exosuit, 92-93 

gestural interface, 97 

interface, 34 

virtual telepresence, 145 

volumetric projection, 78, 86, 89 
The Matrix Revolutions (2003), 5 5 
McLuhan, Marshall, 124 
meaning, organization for adding, 

mechanical controls 

in 1920s and 1930s, 17 

absence in early films, 16-17 

coexistence with other interfaces, 

disappearance in films, 21 -23 

for fine motor control, 24 

mixing with other types, 26 

moods evoked by, 26-27 

trends, 27-28 
mechanical feedback, for industrial age 
users, 17 



mediated coupling, 302-306 
medical interfaces. See also 
assistive medical interfaces; 
autonomous medical interfaces 

assisting birth, 283-284 

basics, 258-259 

diagnosis in future, 274 

evolving technology, 290 

future of testing, 272 

monitoring, usefulness vs. impressive, 

in sci-fi, 258, 290 

volumetric projection for imaging, 87 
medical mysteries, 271 
mediums, multiple, and senses for 

learning, 235 
Melies, Georges, 28 
memorization, 246 
memory. See also brain/consciousness 

deleting, 137-138 
Men in Black (1997) 

death in, 288 

memory deletion, 137 

typefaces in, 36 
Men in Black 2 (2002) 

brain interface, 129 

medical treatment, 276 

screen shape, 50 
menus, 36, 55 
messages, 199 

accepting, 211-212 

communicating, 81 

composing, 199-200 

playback of recorded, 2 01 

recording signal, 200 

sound for urgent attention to, 208 

unique identifier for recipient, 205-206 

volumetric projection for, 81 
metaphors, 19 

spatial, 31 
Metropolis (1927), 17, 64, 218 

brain interface, 128 

communication, 198 

conversational interface, 1 1 

reading from brain, 139 

robots, 182 

tags for videophone, 7-8 

voice interface, 122 
Microgramma typeface, 37 

Kinect, 25, 108 

unsuccessful guides, 192-193 
midwife robot, 281, 283 
mind. See brain/consciousness interfaces 
Mindflex game (Mattel), 155 
Minority Report (2002) 

brain interface, 128,131 

communication, 215 

gestural interface, 57, 95-97, 98, 103 

location awareness, 163-1 64 

precrime scrubber, 107 

reading from brain, 140, 141 

screen display, 51,52 

spyders, 182 

surgery, 278-279 

suspended animation, 265 

volumetric projection, 79 
Mission: Impossible: Ghost Protocol (2011), 

Mission: Impossible (TV show), 276 
Mission to Mars (2000), 45, 51, 52 

typefaces, 38 
MIT Media Lab, 57 
mixed cases for characters, 39 
modeling tasks over time, 232 

communication connection, 212 

in medical interfaces, 260 
moods, mechanical controls 

evoking, 26-27 
Moon (2009), 178 
Mori, Masahiro, 183 
motion graphics, 62-64 
Motorola, 6 
mouse, 30, 102 
multifactor authentication, 1 1 8 
music interfaces, 114-115 
mute function, for audio, 215 




narrative point of view, 1 04-1 07 

choice, 107 
Nass, Clifford, 179 
natural gesture technologies, 25. 

See also gestural interface 
natural user interface, 1 8 
navigation interface 

volumetric projection, 86-87 

in When Worlds Collide, 20 
"neurogenic" interface, 150 
new technologies, getting caught up in, 2 5 
Nintendo Wii, 108 

noncombat psychomotor interface, 231 
noninvasive brain interfaces, 127-131 
nonrectangular computer screens, 

notification of call, 208-210 

object awareness, in augmented reality, 

Oblong, 95 

occlusion, volumetric projection and, 89 
OCR A typeface, 37 
OCR B typeface, 37 
Okuda, Michael, 21, 68 
OnStar, 119 
operator for communication 

connection, 204-205 
orange, in interfaces, 45 
organization, adding meaning 

through, 239-240 
The Outer Limits, "Skin Deep," 303 
overlay reality, 159 

pain, potential from brain-affecting 

interfaces, 152-153 
Pandorum (2009), 265 
paper punch cards, 32 
paralinguistics, 188 
parallax, 166 

patient medical history, 274 
Paycheck (2003), 138 
Pepper's ghost, 76, 87, 88 

forced, for real-world object 
selection, 305 

options for interpretable 
information, 195 

understanding different, 253 
pidgin, gestural, 98-101 
piloting spaceship, AI for, 150-151 
pinch and spread for scaling, 1 01 
Pinker, Steven, 258 
pointing devices, 36 
pointing to select objects, 1 
point of view, for history study, 195 
pornography, 300 
Predator (1987), 173 
presentation tools for learning, 232-236 

in classroom environment, 236 

holodeck, 250 
preventive medicine, 259-260 
print-based typography, 38 
psychomotor practice, 227-231 

holodeck for, 248-250 
purple, in interfaces, 45 
push to move, in gestural interface, 99 
push-to-talk paradigm, 214 

R2-D2, 179, 188, 190 

message playback, 201 
readability, 162, 264 
reading from brain, 138-142 
real-time communication, 199 
real-time visualization of 

childbirth, 284 
recall, recognition vs., 252 
receiving call, 208-214 
recipient, specifying for 

communication, 203-207 
reciprocal influence, 2 
recognition vs. recall, 252 
recorded messages, playback, 201 




as danger indicator, 44 

in interfaces, 43 
Red Planet (2000), 43 
re-embodiment technology, 94 
Reeves, Byron, 179 
reference tools, 236-241 

ease of use vs. error prevention, 239 

holodeck for presenting, 250 
remote connection, for brain 

interfaces, 130-131 
resting position for brain-writing, 1 3 5 
reticle, 172, 174 
revival from death, 284-289 
Robocop (1987) 

augmented reality, 171 

location awareness, 163 

reticle design, 173 
robots, 182-183 

gestures for controlling, 1 6 

medical, 281 

voice interfaces on, 21 
Roddenberry, Gene, 274 

The Making of Star Trek, 275 
Roddenberry, Majel Barrett, 122 
Roomba vacuum robot (iRobot), 1 8 

safe space for learning, 247 

safewords, 303 

sand tables, 11-12 

sans serif typeface, 36,37 


pinch and spread for changing, 1 01 

to gain user attention, 59 
scans in medical interface, 260, 

science fiction, 3-5 

contemporary paradigms and, 1 6 

glow, 40 

inspiration in, 11-13 

reasons to consider, 6-7 

technology inventions, 2 

titles considered as, 5 

using, 310-311 

science fiction interfaces, 31 

World War I and, 18 
screen-based controls, 25 
screen-based typography, 38 
screen interfaces in sci-fi, 43 
scrolling long document in direct 

interface, 102 
Sculleyjohn, 312 
sensitization, 254 
Serenity (2005) 

presentation tools, 235 

sexbots, 297, 298 

volumetric projection, 78 
serifed typefaces, 36 
sex, 292-307 

coupling technologies, 300-306 

interfaces, lack of, 307 

matchmaking, 292-295 

real-world interfaces, 297 

role in sci-fi, 292 

subtle cues for mood, 301 

with technology, 295-300 

virtual, 149-150 
sexbots, 297-298 
sex with technology 

safewords for technologies, 303 

virtual partners, 299 
Sexworld (1978), 300-301 
The Shadow (radio play), 275 
Sherlock Holmes (2009), 311 
shooting in gestural interface, 1 01 
signal for death, 286-289 
Silicon Graphics, 3D File System 

Navigator, 30 
similarities, design lessons from, 9 
Siri (Apple), 119,121,207 
skeuomorphs, 19, 55 
Sleep Dealer (2008), 97,105,106 
Sleeper {191% 296 

medical treatment, 276 
"Snow White" design language (Frog 

Design), 313 
Snyder, Allan, 152 

social hierarchy, volumetric projection 
reinforcement of, 85-86 



social interaction as focus, 21 8 
social norms of humans, 1 8 
social pressure of agent, 1 92 
sonic interfaces, 110-124 

ambient sound, 112 

directional sound, 112-114 

experiential history, 111 

music, 114-115 

sound effects, 110-112 

technologies included, 1 1 

user expectations, 124 

voice, 115-124 

audio communication, 214-217 

for urgent attention to message, 208 
sound effects, 110-112 
Space: 1999 (1975), 48, 56 

audio communication, 215 

medical monitoring, 263, 264-265 

medical scanning, 266 

signal for death, 289 
Spaceballs (1987), 311 
spacefaring in sci-fi, 18 
spaceship, artificial intelligence for 

piloting, 150 
Space Truckers (1996), 301 
spatial memory of user, 3D data and, 62 
spatial metaphor, 31 
spatial sound for nonspatial 

information, 113 
spatial user interface, 30 
specificity, in medical testing, 271 
speculative fiction, 5, 7, 13, 21 
speculative technology, studies of other 

domains, 311, 312 
Stanford University School of 

Medicine, 270 
Star Gate SG-1, 135 
Starship Troopers (1997), 243 

communication, 201 

games for learning, 230 

medical treatment, 279 

presentation tools, 234 

reticle design, 173 

typefaces, 38 

volumetric projection, 79 
StarTAC, 6 
Star Trek 

communicator, 6, 202 

computer conversational 
interfaces, 122, 242-243 

controls in, 23 

engine sounds, 112 

Enterprise, 24 

holodeck, 78, 93, 190, 247-255 

hypospray, 275 

interfaces, 25 

LCARS interface, 68-73 

medical tricorder, 271, 273-274 

presentations, 232 

sound effects, 111 

testing interface, 246 

voice-authorization technology, 1 1 7 

voice control, 123 

warning countdown, 116 
Star Trek (2009) 

glow in, 40 

motion graphics, 62, 63 

signal for death, 287 

testing interfaces, 245, 246-247 

transparent layers of information, 5 3 

voice-identification system, 118 
Star Trek II: The Wrath of Khan (1982) 

testing interface, 246 
Star Trek IV: The Voyage Home (1986) 

communication, 218, 219 

testing as learning, 244 
Star Trek: Deep Space Nine 

brain interface, 146 

LCARS interface, 72 

LCARS interface compared to 
Enterprise, 46 

virtual partners, 299 
Star Trek: Insurrection (1998), 26 
Star Trek: Nemesis (2002), 4 3 
Star Trek: The Next Generation 

brain interface, 128,135-136,151 

brain interface remote 

connection, 130-131, 135-136 

brain-reading technology, 1 5 3 



Star Trek: The Next Generation (continued) 

color coding, 48,49 

communication, 202, 211 

controls, 21, 22 

holodeck, 93, 248, 249, 251-252, 253 

LCARS interface, 48, 68-73 

Lieutenant Data, 182-183 

Lieutenant Data voice mimicry 
ability, 118 

medical monitoring, 267-268 

Thought Maker, 130-131,135-136 
Star Trek: The Original Series, 22, 62, 273 

brain interface, 225 

medical monitoring, 258, 260-261 

medical mysteries, 271 

medical treatment, 275, 277 

typefaces in, 36 
Star Trek: Voyager, 49 

anthropomorphism, 187 

brain interface, 129,150 

holodeck as learning tool, 224, 
250-251, 253 

interfaces, 72 

medical history, 274 

medical mystery, 271 

medical scanning, 266 

medical tricorder, 273 

motion graphics, 63 

TCARS interface, 71 

thought interface, 153 

virtual sex partners, 299 

volumetrically projected doctor, 280, 
Star Wars 

C-3PO, 182, 190 

presentation tools, 232-233, 234 

robotic medical care, 281 

R2-D2, 179,182,188,190 

screen shape, 50 

volumetric projection, 79, 81-82, 83, 
84-85, 86, 88 
Star Wars Episode I: The Phantom Menace 
(1999), 45 

color use, 46, 47 

screen shape, 50 

Star Wars Episode II: Attack of the Clones 

presentation tools, 234 

volumetric projection, 79, 88 

weapon interface, 173 
Star Wars Episode III: Revenge of the Sith 

birth technology, 283 

robotic medical care, 281 

volumetric projection, 79, 83, 
Star Wars Episode IV: A New Hope (1977), 

audio communication, 215 

augmented reality, 158,162 

directional sound, 113 

presentations, 232-233 

psychomotor practice, 228 

R2-D2, 201 

screen shape, 50 

sensor information, 161 
Star Wars Episode V: The Empire Strikes 
Back (1980) 

medical treatment, 279 

robotic medical doctor, 281 

suspended animation, 265 

volumetric projection, 81-82, 86 
Star Wars Episode VI: Return ofthejedi 

presentation tools, 233 
The Stepford Wives (1975), 298 
stereoscopic technologies, 77 
stock-trading services, agency vs. 

autonomy, 190 
stored contacts, 206-207 
Strange Days (1995), 144,306 
Sunshine (2007), 199-200 
Sun, Starfire (1994), 312 
Superman (1978), reference 
technology, 237-238 
Supernova (2000), 42 
surgery, 276-280 
suspended animation, 265 
swiping to dismiss objects, 1 



SWISS 911 Ultra Compressed typeface, 
37, 69 

synchronous communication 
vs. asynchronous, 199-202 
volumetric projection for, 81 

tag cloud, 8 

targeting, in augmented reality, 172, 174 

TCARS interface, in Star Trek: 

Voyager, 71 
teacher input to learning interface, 229 

connection between brain and, 

mimicing alien species, 181 

sex with, 295-300 

number entry system, 206 

ringing, 111 

voice-response support systems, 119 

actual, 148-149 

virtual, 144-148 
teletype terminals, 32 
telexperience, 142-144 
Temporal Computer Access and Retrieval 

System (TCARS), 71 
The Terminator (1984), 183 
Terminator 2: Judgment Day (1991) 

friend or foe assessment, 168-169 

goal awareness, 174,175 

typefaces in, 38 
terrain modeling, 164 

hypotheses, 252 

as learning interface, 24 4 -247 

medical, 260, 271-272 
text-based interfaces, 32-35 
text labels, size of, 37 
text message, recording, 1 9 9 
Things to Come (1936), 51,236 

interfaces, 153-155 

invisibility of, 126 

machines for, 241-244 

manifesting in active subject, 149 
threat assessment, 168-169 
THX-U38 (1971), 295-296 
time, interface use over, 4 
The Time Machine (2002), Vox library 

interface, 191-192 
The Time Machine (Wells, 1895), 4 
time-share systems, 32 
toolbars, 55 

top down, learning lessons from, 7-8, 10 

color use, 46, 47 

revival from death, 285 
Total Recall (1990), 231 

matchmaking, 294 

medical treatment, 276 

volumetric projection, 78 
touching to select objects, 100 
touch-screen technology, 22, 25, 68, 72 

gestural interactions, 94 
training, 205. See also learning 
transcranial magnetic stimulation 

(TMI), 152 
Transformers (2007), 44,45 

sensor display, 161 
translation of languages, 218-220 
translucency, for volumetric 

projection, 88 
transparency in GUIs, 51-54 
transparent displays, 51 
triggers, for preprogrammed effects in 

subject's mind, 132 
Tron: Legacy (2010), 78 
turning objects, in gestural interface, 99 
typography, 36-39, 73 


uncanny valley, 183-185 
Underkoffler, John, 95 
unique identifier for message 

recipient, 205-206 
US Army Topographic Engineering 

Center, 11 



Until the End of the World (1991), 36, 132 

Bounty Bear search program, 181 

brain reading, 143 

memory wiping, 138 

telexperience technology, 1 41 -142 
unwilling subjects, writing to 

brain, 135-138 
user expectations 

for real-world interfaces, 2 

for robots, 185 

for sonic interfaces, 124 
user interfaces, 3 

intervention opportunities, 169 

speculative interface, 310 
user spatial memory, 3D data and, 62 


video communication, 217-218 

feedback on connection, 2 1 2 
video editing machine, 59 
videophone in Metropolis, tags for, 7-8 
viewer's perspective, augmentation 

graphics and, 166 
virtual partners, 299 
virtual reality, 159 

vs. real world, 80,145 
virtual sex, 149-150 
virtual telepresence, 144-148 
vision field, augmented reality in 

peripheral, 162 
visual conventions, for projected 

image, 80 
visual design in interface, 31 
visual hierarchies, 39 
visual interfaces 

basics, 30-31 

elements included, 32 

for detailed picture of future, 73 

graphical user interfaces (GUIs), 36-64 

text-based, 32-35 
visual style of sci-fi interfaces, 64-73 
vital signs, 260, 262, 263, 267, 269 


of gestures, 98-101 

for voice interface commands, 118-121 
voice controls, 25,28 
voice-identification interfaces, 1 1 7-1 1 8 
voice interfaces, 110,115-124 

in anthropomorphism, 186-187 

commands representative of 
action, 121 

control of Star Trek holodeck, 248 

conversational, 121-124 

knowledge of being addressed, 122 

limited command, 118-121 

output, 116-117 

on robots, 21 

vs. gestural, 104 
voiceprint, system check for, 1 1 7 
volume controls for interfaces, absence 

of, 217 
volumetric projection (VP), 76-90 

appearance of, 78-81 

gaze-matching problem, 82-83 

for interpreting, 84 

in medical treatment, 268-269, 279 

navigation with, 86-87 

real-world problems, 87-89 

risk of overuse, 89 

for sex, 295-296 

social hierarchy reinforcement, 85-86 

on Star Trek holodeck, 248 

in Superman, 237 

terminology, 76-78 

uses, 81-87 

visual language of, 81 
Le voyage dans la lune (1902), 16,28 


War Games (1983), 33 

warning countdown, voice output 

for, 116 
waveforms, for data over time, 262-265 
wave, to activate gestural interface, 98 
weapon interface, 172-175 
music as, 114 



website accompanying book, 5 

Weird Science (1985), 293-294 

Wells, H. G., The Time Machine (1895), 4 

Westworld (1973), 298 

"wetware" interface, 105 

Wzerc Worlds Collide (1951), 19, 20 

Wii (Nintendo), 108 

willing subject, writing to brain of, 

WIMP interface elements (windows, 

icons, menus, pointing devices), 36 
vs. command-line for references, 239 
conventions, 57 
windows, 36 
worn devices, for brain interfaces, 

writing to brain, 132-138 
Wurman, Richard Saul, 239 


X2(2003), 38 

Xenotran Mark II Dynamic Sand Table, 1 2 
Xenovision, 12 
X-Aferc (2000), 11, 117 
augmented reality, 172 

yellow, in interfaces, 45 

Zardoz (1974), 215 
zoomorphism, 196 




Thanks to our editor JoAnn Simony and publisher Lou Rosenfeld for their 
patience, mighty expertise, and help getting the book in top shape. 

We'd also like to acknowledge our technical readers for their helpful 
suggestions and insights: Giles Colborne, Rohan Dixit, Michelle Katz, Robert 
Reimann, and Dan Saffer. In addition, Brian David Johnson has been an 
inspiration and supporter of this project since we met him several years ago. 

Thanks to those makers of both sci-fi and real-world designs who gave 
their time and thoughts in panels and interviews (sorry we weren't able to 
include them): Douglas Caldwell, Mark Coleran, Mike Fink, Neil Huxley, 
Dean Kamen, Joe Kosmo, David Lewindowski, Jerry Miller, Michael Ryman, 
Rpin Suwannath, and Lee Weinstein. 

Thanks also to those fans whose impressive attention to detail and 
collaborative effort gave us online references that allowed us to check our 
findings. Wikis like Wookiepedia ( and 
Memory-Alpha ( proved invaluable. 

And of course thanks to those tens of thousands of people who make sci- 
fi movies and television shows and share their visions of the future and 
future technology. 

Lastly, we'd like to thank those who came to our public talks at SXSW, 
MacWorld, dConstruct, and many other events, and shared their thoughts 
and examples about this exceptionally fun and nerdy topic. You've pointed 
us in directions we wouldn't have known to look otherwise, and made us 
think about our assumptions and conclusions. 

I'd like to thank my friends and family who've had to suffer with my talking 
solely about science fiction for five years, and for not making fun of my not 
having a clue what other media has come out in that time. 

—Nathan Shedroff 

Thanks to Ben for the love, support, and patience with my turning almost 
every movie night into a sci-fi night. Thanks to the folks at Cooper for 
support and lunchtime conversations about ideas in progress. Thanks also 
to Mom and Dad, friends and family. 

— Chris Noessel 



Nathan Shedroff is chair of the groundbreaking MBA 
in Design Strategy program at the California College 
of the Arts in San Francisco. This program melds the 
unique principles that design offers business strategy 
with a vision of the future of business as sustainable, 
meaningful, and truly innovative— as well as profitable. 

Nathan is one of the pioneers in experience design 
and has played an important role in the related fields 
of interaction design and information design. He is a serial entrepreneur, 
works in several media, and consults strategically for companies to build 
better, more meaningful experiences for their customers. 

Nathan speaks and teaches internationally and has written extensively on 
design and business issues. He is the author of Experience Design 1.1, and 
co&uXhoxed Making Meaning with two members of Cheskin, a Silicon Valley- 
based strategy consultancy, to explore how companies can create products 
and services specifically to evoke meaning in their audiences and customers. 
Nathan is editor of the Dictionary of Sustainable Management, a website 
and now a printed book. He also maintains an extensive set of resources on 
experience design on his website ( 

Nathan earned a degree in industrial design with an emphasis on 
automobile design from the Art Center College of Design in Pasadena. 
A passion for information design led Nathan to work with Richard Saul 
Wurman at TheUnderstandingBusiness. Later, he co-founded vivid studios, 
a pioneering company in interactive media and one of the first Web services 
firms, vivid's hallmark was helping to establish and validate the field of 
information architecture by training an entire generation of designers in 
the newly emerging Web industry. Nathan was nominated for a Chrysler 
Innovation in Design Award in 1994 and 1999, and a National Design 
Award in 2001. In 2006 Nathan earned an MBA at the Presidio School of 
Management in San Francisco, the only accredited MBA program in the 
United States specializing in sustainable business. 


Christopher Noessel, in his day job as managing 
director at the pioneering interaction design firm 
Cooper, designs products, services, and strategy for 
the health, financial, and consumer domains, among 
others. In his role as practice lead, he helps manage the 
"generator" type of interaction designers, helping them 
build their skills and lead client projects to greatness. 

Christopher has been doing interaction design for 
more than 20 years (longer than we've even been 
calling it that). He co-founded a small interaction design agency where he 
developed interactive exhibitions and environments for museums, and he 
worked as a director of information design at international Web consultancy 
marchFIRST, where he also helped establish the interaction design Center 
of Excellence. 

Christopher was one of the founding graduates of the now-passing-into- 
legend Interaction Design Institute Ivrea in Ivrea, Italy, where his thesis 
project was a comprehensive service design for lifelong learners called Fresh. 
The project was presented at the MLearn conference in London in 2003. He 
has since helped to visualize the future of counterterrorism as a freelancer, 
built prototypes of coming technologies for Microsoft, and designed 
telehealth devices to accommodate the crazy facts of modern health care in 
his role at Cooper. 

Christopher has written for online publications for many years, and was 
first published in print as co-author of the interaction design pattern 
chapter in the textbook edited by Simson Garfinkel, RFID: Applications, 
Security, and Privacy. His Spidey sense goes off at random topics, and this 
has led him to speak at conferences around the world about a wide range 
of things, including interactive narrative, ethnographic user research, 
interaction design, sex-related interactive technologies, free-range learning, 
the Interface Parenthesis and the future of interaction design, and the 
relationship between science fiction and interface design. 

348 About the Authors